Document the new test tools.

5 files changed, 178 insertions, 140 deletions

diff --git a/.spell-dict b/.spell-dict
index 13a9439..ab6b7a5 100644
--- a/.spell-dict
+++ b/.spell-dict
@@ -11,6 +11,7 @@ Blockprocessor
 Blockprocessors
 blockquote
 blockquotes
+boolean
 CamelCase
 Chodarev
 CLI
@@ -18,6 +19,7 @@ CodeHilite
 Cogumbreiro
 convertFile
 CSS
+dedent
 deliminators
 Dmitry
 docdata
diff --git a/docs/change_log/release-2.1.md b/docs/change_log/release-2.1.md
index 30cf0b2..c8fa0e4 100644
--- a/docs/change_log/release-2.1.md
+++ b/docs/change_log/release-2.1.md
@@ -112,7 +112,7 @@ script to accept input on `stdin`.
 The testing framework has been completely rebuilt using the Nose testing
 framework. This provides a number of benefits including the ability to better
 test the built-in extensions and other options available to change the parsing
-behavior. See the [Test Suite](../test_suite.md) documentation for details.
+behavior. See the Test Suite documentation for details.
 
 Various bug fixes have been made, which are too numerous to list here. See the
 [commit log](https://github.com/Python-Markdown/markdown/commits/master) for a
diff --git a/docs/test_suite.md b/docs/test_suite.md
deleted file mode 100644
index 9ead4c6..0000000
--- a/docs/test_suite.md
+++ /dev/null
@@ -1,138 +0,0 @@
-title:      Test Suite
-prev_title: Extension API
-prev_url:   extensions/api.html
-next_title: Change Log
-next_url:   change_log.html
-
-# Test Suite
-
-Python-Markdown comes with a test suite which uses the [Nose] testing
-framework and [YAML]. The test suite primarily serves to ensure that new bugs
-are not introduced as existing bugs are patched or new features are added. It
-also allows Python-Markdown to be tested with the tests from other
-implementations such as John Gruber's [Perl] implementation or Michel
-Fortin's [PHP] implementation.
-
-The test suite can be run by calling the `run_tests.py` command at the root of
-the distribution tarball or by calling the `nosetests` command directly. Either
-way, Nose will need to be installed on your system first (run `easy_install
-nose`). Any standard nosetests configuration options can be passed in on the command
-line (i.e.: verbosity level or use of a plugin like coverage).
-
-Additionally, a nicely formatted HTML report of all output is written to a
-temporary file in `test-output.html`. Open the file in a browser to view
-the report.
-
-A `tox.ini` file is also provided, so [tox] can be used to automatically create
-virtual environments, install all testing dependencies and run the tests on
-each supported Python version. See the wiki for instructions on
-[setting up a testing environment] to use tox.
-
-The test suite contains two kinds of tests: Markdown Syntax Tests and Unit
-Tests.
-
-## Markdown Syntax Tests
-
-The Syntax Tests are in the various directories contained within the 'tests'
-directory of the packaged tarball. Each test consists of a matching pair of text
-and HTML files. The text file contains a snippet of Markdown source text
-formatted for a specific syntax feature and the HTML file contains the expected
-HTML output of that snippet. When the test suite is run, each text file is run
-through Markdown and the output is compared with the HTML file as a separate
-Unit Test.
-
-In fact, this is the primary reason for using Nose, it gives us an easy way to
-treat each of these tests as a separate unit test which is reported on
-separately. Additionally, with the help of a couple custom Nose plugins which
-are included with the Markdown Test Suite, we are able to get back an easy to
-read diff of the actual output compared to expected output when a test fails.
-
-Here is some sample output with a test that is failing because of some
-insignificant white space differences:
-
-    $ ./run-tests.py
-    ..........................................................M...........
-    ............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
-    SSSSSSSSSS.................S..........................................
-    .........
-    ======================================================================
-    MarkdownSyntaxError: TestSyntax: "misc/lists3"
-    ----------------------------------------------------------------------
-    MarkdownSyntaxError: Output from "/home/waylan/code/python-markdown/te
-    sts/misc/lists3.txt" failed to match expected output.
-
-    --- /home/waylan/code/python-markdown/tests/misc/lists3.html
-    +++ actual_output.html
-    @@ -1,5 +1,5 @@
-     <ul>
-     <li>blah blah blah
-    -sdf asdf asdf asdf asdf
-    -asda asdf asdfasd</li>
-    +    sdf asdf asdf asdf asdf
-    +    asda asdf asdfasd</li>
-     </ul>
-
-    ----------------------------------------------------------------------
-    Ran 219 tests in 7.698s
-
-    FAILED (MarkdownSyntaxError=1, SKIP=53)
-
-Note that 219 tests were run, one of which failed with a `MarkdownSyntaxError`.
-Only Markdown Syntax Tests should fail with a `MarkdownSyntaxError`. Nose then
-formats the error reports for `MarkdownSyntaxError`s so that they only include
-useful information. Namely the text file which failed and a unified diff showing
-the failure. Without the plugin, you would also get a useless traceback showing
-how the code stepped through the test framework, but nothing about how Markdown
-actually ran.
-
-If, on the other hand, a Syntax Test failed because some other exception gets
-raised by either Markdown or the test suite, then that would be reported as per
-a normal unit test failure with the appropriate traceback for debugging
-purposes.
-
-### Syntax Test Configuration Settings
-
-The other thing to note about the above example is that 53 tests were skipped.
-Those tests have been explicitly configured to be skipped as they are primarily
-tests from either PHP or Perl which are known to fail for various reasons. In
-fact, a number of different configuration settings can be set for any specific
-test.
-
-Each Syntax Test directory contains a `test.cfg` file in the [YAML] format. The
-file may contain a separate section for each text file named exactly as the file
-is named minus the file extension (i.e.; the section for a test in `foo.txt`
-would be `foo`). All settings are optional. Default settings for the entire
-directory can be set under the `DEFAULT` section (must be all caps). Any
-settings under a specific file section will override anything in the
-`DEFAULT` section for that specific test only.
-
-Below are the configuration options available and the defaults used when they
-are not explicitly set.
-
-* `normalize`: Switches white space normalization of the test output on or off.
-  Defaults to `False` (off). Note: This requires that [PyTidyLib] be installed
-  on the system. Otherwise the test will be skipped, regardless of any other
-  settings.
-* `skip`: Switches skipping of the test on and off. Defaults to `False` (off).
-* `input_ext`: Extension of input file. Defaults to `.txt`. Useful for tests
-  from other implementations.
-* `output_ext`: Extension of output file. Defaults to `.html`. Useful for tests
-  from other implementations.
-* Any keyword argument accepted by the Markdown class. If not set, Markdown's
-  defaults are used.
-
-## Unit Tests
-
-Unit Tests are used as regression tests for Python-Markdown's API.
-All Unit Tests shipped with Python-Markdown are standard Python Unit Tests and
-are all contained in `tests/test_apis.py` and `tests/test_extensions.py`.
-Standard discovery methods are used to find and run the tests. Therefore, when
-writing new tests, those standards and naming conventions should be followed.
-
-[Nose]: http://somethingaboutorange.com/mrl/projects/nose/
-[Perl]: http://daringfireball.net/projects/markdown/
-[PHP]: http://michelf.com/projects/php-markdown/
-[PyTidyLib]: http://countergram.com/open-source/pytidylib/
-[tox]: http://testrun.org/tox/latest/
-[setting up a testing environment]: https://github.com/Python-Markdown/markdown/wiki/Test-Environment-Setup
-[YAML]: http://yaml.org/
diff --git a/docs/test_tools.md b/docs/test_tools.md
new file mode 100644
index 0000000..c2d5951
--- /dev/null
+++ b/docs/test_tools.md
@@ -0,0 +1,174 @@
+title: Test Tools
+
+# Test Tools
+
+Python-Markdown provides some testing tools which simplify testing actual
+Markdown output again expected output. The tools are built on the Python
+standard  library [`unittest`][unittest]. Therefore, no additional libraries are
+required. While Python-Markdown uses the tools for its own tests, they were
+designed and built so that third party extensions could use them as well.
+Therefore, the tools are importable from `markdown.test_tools`.
+
+The test tools include two different `unittest.TestCase` subclasses:
+`markdown.test_tools.TestCase` and `markdown.test_tools.LegacyTestCase`.
+
+## markdown.test_tools.TestCase
+
+The `markdown.test_tools.TestCase` class is a `unittest.TestCase` subclass with
+a few additional helpers to make testing Markdown output easier.
+
+Properties
+: `default_kwargs`: A `dict` of keywords to pass to Markdown for each
+test. The defaults can be overridden on individual tests.
+
+Methods
+: `assertMarkdownRenders`: accepts the source text, the expected output,
+  and any keywords to pass to Markdown. The `default_kwargs` defined on the
+  class are used except where overridden by keyword arguments. The output and
+  expected output are passed to `TestCase.assertMultiLineEqual`. An
+  `AssertionError` is raised with a diff if the actual output does not equal the
+  expected output.
+
+: `dedent`: Dedent triple-quoted strings.
+
+In all other respects, `markdown.test_tools.TestCase` behaves as
+`unittest.TestCase`. In fact, `assertMarkdownRenders` tests could be mixed with
+other `unittest` style tests within the same test class.
+
+An example Markdown test might look like this:
+
+```python
+from markdown.test_tools import TestCase
+
+class TestHr(TestCase):
+    def test_hr_before_paragraph(self):
+        self.assertMarkdownRenders(
+            # The Markdown source text used as input
+            self.dedent(
+                """
+                ***
+                An HR followed by a paragraph with no blank line.
+                """
+            ),
+            # The expected HTML output
+            self.dedent(
+                """
+                <hr>
+                <p>An HR followed by a paragraph with no blank line.</p>
+                """
+            ),
+            # Other keyword arguments to pass to `markdown.markdown`
+            output_format='html'
+        )
+```
+
+## markdown.test_tools.LegacyTestCase
+
+In the past Python-Markdown exclusively used file-based tests. Many of those
+tests still exist in Python-Markdown's test suite, including the test files from
+the [reference implementation][perl] (`markdown.pl`) and [PHP Markdown][PHP].
+Each test consists of a matching pair of text and HTML files. The text file
+contains a snippet of Markdown source text formatted for a specific syntax
+feature and the HTML file contains the expected HTML output of that snippet.
+When the test suite is run, each text file is run through Markdown and the
+output is compared with the HTML file as a separate unit test. When a test
+fails, the error report includes a diff of the expected output compared to the
+actual output to easily identify any problems.
+
+A separate `markdown.test_tools.LegacyTestCase` subclass must be created for
+each directory of test files. Various properties can be defined within the
+subclass to point to a directory of text-based test files and define various
+behaviors/defaults for those tests. The following properties are supported:
+
+* `location`: A path to the directory of test files. An absolute path is
+  preferred.
+* `exclude`: A list of tests to skip. Each test name should comprise of a
+  file name without an extension.
+* `normalize`: A boolean value indicating if the HTML should be normalized.
+  Default: `False`. Note: Normalization of HTML requires that [PyTidyLib] be
+  installed on the system. If PyTidyLib is not installed and `normalize` is set
+  to `True`, then the test will be skipped, regardless of any other settings.
+* `input_ext`: A string containing the file extension of input files.
+  Default: `.txt`.
+* `output_ext`: A string containing the file extension of expected output files.
+  Default: `html`.
+* `default_kwargs`: A `markdown.test_tools.Kwargs` instance which stores the
+  default set of keyword arguments for all test files in the directory.
+
+In addition, properties can be defined for each individual set of test files
+within the directory. The property should be given the name of the file without
+the file extension. Any spaces and dashes in the file name should be replaced
+with underscores. The value of the property should be a
+`markdown.test_tools.Kwargs` instance which contains the keyword arguments that
+should be passed to `markdown.markdown` for that test file. The keyword
+arguments will "update" the `default_kwargs`.
+
+When the class instance is created during a test run, it will walk the given
+directory and create a separate unit test for each set of test files using the
+naming scheme: `test_filename`. One unit test will be run for each set of input
+and output files.
+
+The definition of an example set of tests might look like this:
+
+```python
+from markdown.test_tools import LegacyTestCase, Kwargs
+import os
+
+# Get location of this file and use to find text file dirs.
+parent_test_dir = os.path.abspath(os.path.dirname(__file__))
+
+
+class TestFoo(LegacyTestCase):
+    # Define location of text file directory. In this case, the directory is
+    # named "foo" and is in the same parent directory as this file.
+    location = os.path.join(parent_test_dir, 'foo')
+    # Define default keyword arguments. In this case, unless specified
+    # differently, all tests should use the output format "html".
+    default_kwargs = Kwargs(output_format='html')
+
+    # The "xhtml" test should override the output format and use "xhtml".
+    xhtml = Kwargs(output_format='xhtml')
+
+    # The "toc" test should use the "toc" extension with a custom permalink
+    # setting.
+    toc = Kwargs(
+        extensions=['markdown.extensions.toc'],
+        extension_configs={'markdown.extensions.toc': {'permalink': "[link]"}}
+    )
+```
+
+Note that in the above example, the text file directory may contain many more
+text-based test files than `xhtml` (`xhtml.txt` and `xhtml.html`) and `toc`
+(`toc.txt` and `toc.html`). As long as each set of files exists as a pair, a
+test will be created and run for each of them. Only the `xhtml` and `toc` tests
+needed to be specifically identified as they had specific, non-default settings
+which needed to be defined.
+
+## Running Python-Markdown's Tests
+
+As all of the tests for the `markdown` library are unit tests, standard
+`unittest` methods of calling tests can be used. For example, to run all of
+Python-Markdown's tests, from the root of the git repository, run the following
+command:
+
+```sh
+python -m unittest discover tests
+```
+
+That simple command will search everything in the `tests` directory and it's
+sub-directories and run all `unittest` tests that it finds, including
+`unittest.TestCase`, `markdown.test_tools.TestCase`, and
+`markdown.test_tools.LegacyTestCase` subclasses. Normal `unittest` discovery
+rules apply.
+
+Python-Markdown's git repository also includes a `tox.ini` file, so [tox] can be
+used to automate the creation of virtual environments, installation of all
+testing dependencies and running of the tests on each supported Python version.
+See the wiki for instructions on [setting up a testing environment] to use tox.
+
+[unittest]: https://docs.python.org/3/library/unittest.html
+[Perl]: http://daringfireball.net/projects/markdown/
+[PHP]: http://michelf.com/projects/php-markdown/
+[PyTidyLib]: http://countergram.com/open-source/pytidylib/
+[tox]: http://testrun.org/tox/latest/
+[setting up a testing environment]: https://github.com/Python-Markdown/markdown/wiki/Test-Environment-Setup
diff --git a/mkdocs.yml b/mkdocs.yml
index 8d2bcea..a732af4 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -37,7 +37,7 @@ pages:
       - Tables: extensions/tables.md
       - WikiLinks: extensions/wikilinks.md
   - Extension API: extensions/api.md
-  - Test Suite: test_suite.md
+  - Test Tools: test_tools.md
   - Change Log: change_log/index.md
   - Release Notes for v.2.6: change_log/release-2.6.md
   - Release Notes for v.2.5: change_log/release-2.5.md