diff options
| -rw-r--r-- | .spell-dict | 2 | ||||
| -rw-r--r-- | docs/change_log/release-2.1.md | 2 | ||||
| -rw-r--r-- | docs/test_suite.md | 138 | ||||
| -rw-r--r-- | docs/test_tools.md | 174 | ||||
| -rw-r--r-- | mkdocs.yml | 2 |
5 files changed, 178 insertions, 140 deletions
diff --git a/.spell-dict b/.spell-dict index 13a9439..ab6b7a5 100644 --- a/.spell-dict +++ b/.spell-dict @@ -11,6 +11,7 @@ Blockprocessor Blockprocessors blockquote blockquotes +boolean CamelCase Chodarev CLI @@ -18,6 +19,7 @@ CodeHilite Cogumbreiro convertFile CSS +dedent deliminators Dmitry docdata diff --git a/docs/change_log/release-2.1.md b/docs/change_log/release-2.1.md index 30cf0b2..c8fa0e4 100644 --- a/docs/change_log/release-2.1.md +++ b/docs/change_log/release-2.1.md @@ -112,7 +112,7 @@ script to accept input on `stdin`. The testing framework has been completely rebuilt using the Nose testing framework. This provides a number of benefits including the ability to better test the built-in extensions and other options available to change the parsing -behavior. See the [Test Suite](../test_suite.md) documentation for details. +behavior. See the Test Suite documentation for details. Various bug fixes have been made, which are too numerous to list here. See the [commit log](https://github.com/Python-Markdown/markdown/commits/master) for a diff --git a/docs/test_suite.md b/docs/test_suite.md deleted file mode 100644 index 9ead4c6..0000000 --- a/docs/test_suite.md +++ /dev/null @@ -1,138 +0,0 @@ -title: Test Suite -prev_title: Extension API -prev_url: extensions/api.html -next_title: Change Log -next_url: change_log.html - -# Test Suite - -Python-Markdown comes with a test suite which uses the [Nose] testing -framework and [YAML]. The test suite primarily serves to ensure that new bugs -are not introduced as existing bugs are patched or new features are added. It -also allows Python-Markdown to be tested with the tests from other -implementations such as John Gruber's [Perl] implementation or Michel -Fortin's [PHP] implementation. - -The test suite can be run by calling the `run_tests.py` command at the root of -the distribution tarball or by calling the `nosetests` command directly. Either -way, Nose will need to be installed on your system first (run `easy_install -nose`). Any standard nosetests configuration options can be passed in on the command -line (i.e.: verbosity level or use of a plugin like coverage). - -Additionally, a nicely formatted HTML report of all output is written to a -temporary file in `test-output.html`. Open the file in a browser to view -the report. - -A `tox.ini` file is also provided, so [tox] can be used to automatically create -virtual environments, install all testing dependencies and run the tests on -each supported Python version. See the wiki for instructions on -[setting up a testing environment] to use tox. - -The test suite contains two kinds of tests: Markdown Syntax Tests and Unit -Tests. - -## Markdown Syntax Tests - -The Syntax Tests are in the various directories contained within the 'tests' -directory of the packaged tarball. Each test consists of a matching pair of text -and HTML files. The text file contains a snippet of Markdown source text -formatted for a specific syntax feature and the HTML file contains the expected -HTML output of that snippet. When the test suite is run, each text file is run -through Markdown and the output is compared with the HTML file as a separate -Unit Test. - -In fact, this is the primary reason for using Nose, it gives us an easy way to -treat each of these tests as a separate unit test which is reported on -separately. Additionally, with the help of a couple custom Nose plugins which -are included with the Markdown Test Suite, we are able to get back an easy to -read diff of the actual output compared to expected output when a test fails. - -Here is some sample output with a test that is failing because of some -insignificant white space differences: - - $ ./run-tests.py - ..........................................................M........... - ............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS - SSSSSSSSSS.................S.......................................... - ......... - ====================================================================== - MarkdownSyntaxError: TestSyntax: "misc/lists3" - ---------------------------------------------------------------------- - MarkdownSyntaxError: Output from "/home/waylan/code/python-markdown/te - sts/misc/lists3.txt" failed to match expected output. - - --- /home/waylan/code/python-markdown/tests/misc/lists3.html - +++ actual_output.html - @@ -1,5 +1,5 @@ - <ul> - <li>blah blah blah - -sdf asdf asdf asdf asdf - -asda asdf asdfasd</li> - + sdf asdf asdf asdf asdf - + asda asdf asdfasd</li> - </ul> - - ---------------------------------------------------------------------- - Ran 219 tests in 7.698s - - FAILED (MarkdownSyntaxError=1, SKIP=53) - -Note that 219 tests were run, one of which failed with a `MarkdownSyntaxError`. -Only Markdown Syntax Tests should fail with a `MarkdownSyntaxError`. Nose then -formats the error reports for `MarkdownSyntaxError`s so that they only include -useful information. Namely the text file which failed and a unified diff showing -the failure. Without the plugin, you would also get a useless traceback showing -how the code stepped through the test framework, but nothing about how Markdown -actually ran. - -If, on the other hand, a Syntax Test failed because some other exception gets -raised by either Markdown or the test suite, then that would be reported as per -a normal unit test failure with the appropriate traceback for debugging -purposes. - -### Syntax Test Configuration Settings - -The other thing to note about the above example is that 53 tests were skipped. -Those tests have been explicitly configured to be skipped as they are primarily -tests from either PHP or Perl which are known to fail for various reasons. In -fact, a number of different configuration settings can be set for any specific -test. - -Each Syntax Test directory contains a `test.cfg` file in the [YAML] format. The -file may contain a separate section for each text file named exactly as the file -is named minus the file extension (i.e.; the section for a test in `foo.txt` -would be `foo`). All settings are optional. Default settings for the entire -directory can be set under the `DEFAULT` section (must be all caps). Any -settings under a specific file section will override anything in the -`DEFAULT` section for that specific test only. - -Below are the configuration options available and the defaults used when they -are not explicitly set. - -* `normalize`: Switches white space normalization of the test output on or off. - Defaults to `False` (off). Note: This requires that [PyTidyLib] be installed - on the system. Otherwise the test will be skipped, regardless of any other - settings. -* `skip`: Switches skipping of the test on and off. Defaults to `False` (off). -* `input_ext`: Extension of input file. Defaults to `.txt`. Useful for tests - from other implementations. -* `output_ext`: Extension of output file. Defaults to `.html`. Useful for tests - from other implementations. -* Any keyword argument accepted by the Markdown class. If not set, Markdown's - defaults are used. - -## Unit Tests - -Unit Tests are used as regression tests for Python-Markdown's API. -All Unit Tests shipped with Python-Markdown are standard Python Unit Tests and -are all contained in `tests/test_apis.py` and `tests/test_extensions.py`. -Standard discovery methods are used to find and run the tests. Therefore, when -writing new tests, those standards and naming conventions should be followed. - -[Nose]: http://somethingaboutorange.com/mrl/projects/nose/ -[Perl]: http://daringfireball.net/projects/markdown/ -[PHP]: http://michelf.com/projects/php-markdown/ -[PyTidyLib]: http://countergram.com/open-source/pytidylib/ -[tox]: http://testrun.org/tox/latest/ -[setting up a testing environment]: https://github.com/Python-Markdown/markdown/wiki/Test-Environment-Setup -[YAML]: http://yaml.org/ diff --git a/docs/test_tools.md b/docs/test_tools.md new file mode 100644 index 0000000..c2d5951 --- /dev/null +++ b/docs/test_tools.md @@ -0,0 +1,174 @@ +title: Test Tools + +# Test Tools + +Python-Markdown provides some testing tools which simplify testing actual +Markdown output again expected output. The tools are built on the Python +standard library [`unittest`][unittest]. Therefore, no additional libraries are +required. While Python-Markdown uses the tools for its own tests, they were +designed and built so that third party extensions could use them as well. +Therefore, the tools are importable from `markdown.test_tools`. + +The test tools include two different `unittest.TestCase` subclasses: +`markdown.test_tools.TestCase` and `markdown.test_tools.LegacyTestCase`. + +## markdown.test_tools.TestCase + +The `markdown.test_tools.TestCase` class is a `unittest.TestCase` subclass with +a few additional helpers to make testing Markdown output easier. + +Properties +: `default_kwargs`: A `dict` of keywords to pass to Markdown for each +test. The defaults can be overridden on individual tests. + +Methods +: `assertMarkdownRenders`: accepts the source text, the expected output, + and any keywords to pass to Markdown. The `default_kwargs` defined on the + class are used except where overridden by keyword arguments. The output and + expected output are passed to `TestCase.assertMultiLineEqual`. An + `AssertionError` is raised with a diff if the actual output does not equal the + expected output. + +: `dedent`: Dedent triple-quoted strings. + +In all other respects, `markdown.test_tools.TestCase` behaves as +`unittest.TestCase`. In fact, `assertMarkdownRenders` tests could be mixed with +other `unittest` style tests within the same test class. + +An example Markdown test might look like this: + +```python +from markdown.test_tools import TestCase + +class TestHr(TestCase): + def test_hr_before_paragraph(self): + self.assertMarkdownRenders( + # The Markdown source text used as input + self.dedent( + """ + *** + An HR followed by a paragraph with no blank line. + """ + ), + # The expected HTML output + self.dedent( + """ + <hr> + <p>An HR followed by a paragraph with no blank line.</p> + """ + ), + # Other keyword arguments to pass to `markdown.markdown` + output_format='html' + ) +``` + +## markdown.test_tools.LegacyTestCase + +In the past Python-Markdown exclusively used file-based tests. Many of those +tests still exist in Python-Markdown's test suite, including the test files from +the [reference implementation][perl] (`markdown.pl`) and [PHP Markdown][PHP]. +Each test consists of a matching pair of text and HTML files. The text file +contains a snippet of Markdown source text formatted for a specific syntax +feature and the HTML file contains the expected HTML output of that snippet. +When the test suite is run, each text file is run through Markdown and the +output is compared with the HTML file as a separate unit test. When a test +fails, the error report includes a diff of the expected output compared to the +actual output to easily identify any problems. + +A separate `markdown.test_tools.LegacyTestCase` subclass must be created for +each directory of test files. Various properties can be defined within the +subclass to point to a directory of text-based test files and define various +behaviors/defaults for those tests. The following properties are supported: + +* `location`: A path to the directory of test files. An absolute path is + preferred. +* `exclude`: A list of tests to skip. Each test name should comprise of a + file name without an extension. +* `normalize`: A boolean value indicating if the HTML should be normalized. + Default: `False`. Note: Normalization of HTML requires that [PyTidyLib] be + installed on the system. If PyTidyLib is not installed and `normalize` is set + to `True`, then the test will be skipped, regardless of any other settings. +* `input_ext`: A string containing the file extension of input files. + Default: `.txt`. +* `output_ext`: A string containing the file extension of expected output files. + Default: `html`. +* `default_kwargs`: A `markdown.test_tools.Kwargs` instance which stores the + default set of keyword arguments for all test files in the directory. + +In addition, properties can be defined for each individual set of test files +within the directory. The property should be given the name of the file without +the file extension. Any spaces and dashes in the file name should be replaced +with underscores. The value of the property should be a +`markdown.test_tools.Kwargs` instance which contains the keyword arguments that +should be passed to `markdown.markdown` for that test file. The keyword +arguments will "update" the `default_kwargs`. + +When the class instance is created during a test run, it will walk the given +directory and create a separate unit test for each set of test files using the +naming scheme: `test_filename`. One unit test will be run for each set of input +and output files. + +The definition of an example set of tests might look like this: + +```python +from markdown.test_tools import LegacyTestCase, Kwargs +import os + +# Get location of this file and use to find text file dirs. +parent_test_dir = os.path.abspath(os.path.dirname(__file__)) + + +class TestFoo(LegacyTestCase): + # Define location of text file directory. In this case, the directory is + # named "foo" and is in the same parent directory as this file. + location = os.path.join(parent_test_dir, 'foo') + # Define default keyword arguments. In this case, unless specified + # differently, all tests should use the output format "html". + default_kwargs = Kwargs(output_format='html') + + # The "xhtml" test should override the output format and use "xhtml". + xhtml = Kwargs(output_format='xhtml') + + # The "toc" test should use the "toc" extension with a custom permalink + # setting. + toc = Kwargs( + extensions=['markdown.extensions.toc'], + extension_configs={'markdown.extensions.toc': {'permalink': "[link]"}} + ) +``` + +Note that in the above example, the text file directory may contain many more +text-based test files than `xhtml` (`xhtml.txt` and `xhtml.html`) and `toc` +(`toc.txt` and `toc.html`). As long as each set of files exists as a pair, a +test will be created and run for each of them. Only the `xhtml` and `toc` tests +needed to be specifically identified as they had specific, non-default settings +which needed to be defined. + +## Running Python-Markdown's Tests + +As all of the tests for the `markdown` library are unit tests, standard +`unittest` methods of calling tests can be used. For example, to run all of +Python-Markdown's tests, from the root of the git repository, run the following +command: + +```sh +python -m unittest discover tests +``` + +That simple command will search everything in the `tests` directory and it's +sub-directories and run all `unittest` tests that it finds, including +`unittest.TestCase`, `markdown.test_tools.TestCase`, and +`markdown.test_tools.LegacyTestCase` subclasses. Normal `unittest` discovery +rules apply. + +Python-Markdown's git repository also includes a `tox.ini` file, so [tox] can be +used to automate the creation of virtual environments, installation of all +testing dependencies and running of the tests on each supported Python version. +See the wiki for instructions on [setting up a testing environment] to use tox. + +[unittest]: https://docs.python.org/3/library/unittest.html +[Perl]: http://daringfireball.net/projects/markdown/ +[PHP]: http://michelf.com/projects/php-markdown/ +[PyTidyLib]: http://countergram.com/open-source/pytidylib/ +[tox]: http://testrun.org/tox/latest/ +[setting up a testing environment]: https://github.com/Python-Markdown/markdown/wiki/Test-Environment-Setup @@ -37,7 +37,7 @@ pages: - Tables: extensions/tables.md - WikiLinks: extensions/wikilinks.md - Extension API: extensions/api.md - - Test Suite: test_suite.md + - Test Tools: test_tools.md - Change Log: change_log/index.md - Release Notes for v.2.6: change_log/release-2.6.md - Release Notes for v.2.5: change_log/release-2.5.md |