diff options
Diffstat (limited to 'docs/test_tools.md')
| -rw-r--r-- | docs/test_tools.md | 174 |
1 files changed, 174 insertions, 0 deletions
diff --git a/docs/test_tools.md b/docs/test_tools.md new file mode 100644 index 0000000..c2d5951 --- /dev/null +++ b/docs/test_tools.md @@ -0,0 +1,174 @@ +title: Test Tools + +# Test Tools + +Python-Markdown provides some testing tools which simplify testing actual +Markdown output again expected output. The tools are built on the Python +standard library [`unittest`][unittest]. Therefore, no additional libraries are +required. While Python-Markdown uses the tools for its own tests, they were +designed and built so that third party extensions could use them as well. +Therefore, the tools are importable from `markdown.test_tools`. + +The test tools include two different `unittest.TestCase` subclasses: +`markdown.test_tools.TestCase` and `markdown.test_tools.LegacyTestCase`. + +## markdown.test_tools.TestCase + +The `markdown.test_tools.TestCase` class is a `unittest.TestCase` subclass with +a few additional helpers to make testing Markdown output easier. + +Properties +: `default_kwargs`: A `dict` of keywords to pass to Markdown for each +test. The defaults can be overridden on individual tests. + +Methods +: `assertMarkdownRenders`: accepts the source text, the expected output, + and any keywords to pass to Markdown. The `default_kwargs` defined on the + class are used except where overridden by keyword arguments. The output and + expected output are passed to `TestCase.assertMultiLineEqual`. An + `AssertionError` is raised with a diff if the actual output does not equal the + expected output. + +: `dedent`: Dedent triple-quoted strings. + +In all other respects, `markdown.test_tools.TestCase` behaves as +`unittest.TestCase`. In fact, `assertMarkdownRenders` tests could be mixed with +other `unittest` style tests within the same test class. + +An example Markdown test might look like this: + +```python +from markdown.test_tools import TestCase + +class TestHr(TestCase): + def test_hr_before_paragraph(self): + self.assertMarkdownRenders( + # The Markdown source text used as input + self.dedent( + """ + *** + An HR followed by a paragraph with no blank line. + """ + ), + # The expected HTML output + self.dedent( + """ + <hr> + <p>An HR followed by a paragraph with no blank line.</p> + """ + ), + # Other keyword arguments to pass to `markdown.markdown` + output_format='html' + ) +``` + +## markdown.test_tools.LegacyTestCase + +In the past Python-Markdown exclusively used file-based tests. Many of those +tests still exist in Python-Markdown's test suite, including the test files from +the [reference implementation][perl] (`markdown.pl`) and [PHP Markdown][PHP]. +Each test consists of a matching pair of text and HTML files. The text file +contains a snippet of Markdown source text formatted for a specific syntax +feature and the HTML file contains the expected HTML output of that snippet. +When the test suite is run, each text file is run through Markdown and the +output is compared with the HTML file as a separate unit test. When a test +fails, the error report includes a diff of the expected output compared to the +actual output to easily identify any problems. + +A separate `markdown.test_tools.LegacyTestCase` subclass must be created for +each directory of test files. Various properties can be defined within the +subclass to point to a directory of text-based test files and define various +behaviors/defaults for those tests. The following properties are supported: + +* `location`: A path to the directory of test files. An absolute path is + preferred. +* `exclude`: A list of tests to skip. Each test name should comprise of a + file name without an extension. +* `normalize`: A boolean value indicating if the HTML should be normalized. + Default: `False`. Note: Normalization of HTML requires that [PyTidyLib] be + installed on the system. If PyTidyLib is not installed and `normalize` is set + to `True`, then the test will be skipped, regardless of any other settings. +* `input_ext`: A string containing the file extension of input files. + Default: `.txt`. +* `output_ext`: A string containing the file extension of expected output files. + Default: `html`. +* `default_kwargs`: A `markdown.test_tools.Kwargs` instance which stores the + default set of keyword arguments for all test files in the directory. + +In addition, properties can be defined for each individual set of test files +within the directory. The property should be given the name of the file without +the file extension. Any spaces and dashes in the file name should be replaced +with underscores. The value of the property should be a +`markdown.test_tools.Kwargs` instance which contains the keyword arguments that +should be passed to `markdown.markdown` for that test file. The keyword +arguments will "update" the `default_kwargs`. + +When the class instance is created during a test run, it will walk the given +directory and create a separate unit test for each set of test files using the +naming scheme: `test_filename`. One unit test will be run for each set of input +and output files. + +The definition of an example set of tests might look like this: + +```python +from markdown.test_tools import LegacyTestCase, Kwargs +import os + +# Get location of this file and use to find text file dirs. +parent_test_dir = os.path.abspath(os.path.dirname(__file__)) + + +class TestFoo(LegacyTestCase): + # Define location of text file directory. In this case, the directory is + # named "foo" and is in the same parent directory as this file. + location = os.path.join(parent_test_dir, 'foo') + # Define default keyword arguments. In this case, unless specified + # differently, all tests should use the output format "html". + default_kwargs = Kwargs(output_format='html') + + # The "xhtml" test should override the output format and use "xhtml". + xhtml = Kwargs(output_format='xhtml') + + # The "toc" test should use the "toc" extension with a custom permalink + # setting. + toc = Kwargs( + extensions=['markdown.extensions.toc'], + extension_configs={'markdown.extensions.toc': {'permalink': "[link]"}} + ) +``` + +Note that in the above example, the text file directory may contain many more +text-based test files than `xhtml` (`xhtml.txt` and `xhtml.html`) and `toc` +(`toc.txt` and `toc.html`). As long as each set of files exists as a pair, a +test will be created and run for each of them. Only the `xhtml` and `toc` tests +needed to be specifically identified as they had specific, non-default settings +which needed to be defined. + +## Running Python-Markdown's Tests + +As all of the tests for the `markdown` library are unit tests, standard +`unittest` methods of calling tests can be used. For example, to run all of +Python-Markdown's tests, from the root of the git repository, run the following +command: + +```sh +python -m unittest discover tests +``` + +That simple command will search everything in the `tests` directory and it's +sub-directories and run all `unittest` tests that it finds, including +`unittest.TestCase`, `markdown.test_tools.TestCase`, and +`markdown.test_tools.LegacyTestCase` subclasses. Normal `unittest` discovery +rules apply. + +Python-Markdown's git repository also includes a `tox.ini` file, so [tox] can be +used to automate the creation of virtual environments, installation of all +testing dependencies and running of the tests on each supported Python version. +See the wiki for instructions on [setting up a testing environment] to use tox. + +[unittest]: https://docs.python.org/3/library/unittest.html +[Perl]: http://daringfireball.net/projects/markdown/ +[PHP]: http://michelf.com/projects/php-markdown/ +[PyTidyLib]: http://countergram.com/open-source/pytidylib/ +[tox]: http://testrun.org/tox/latest/ +[setting up a testing environment]: https://github.com/Python-Markdown/markdown/wiki/Test-Environment-Setup |