Renamed *.txt -> *.md in docs.

1 files changed, 611 insertions, 0 deletions

diff --git a/docs/writing_extensions.md b/docs/writing_extensions.md
new file mode 100644
index 0000000..2ecd4c9
--- /dev/null
+++ b/docs/writing_extensions.md
@@ -0,0 +1,611 @@
+Writing Extensions for Python-Markdown
+======================================
+
+Overview
+--------
+
+Python-Markdown includes an API for extension writers to plug their own 
+custom functionality and/or syntax into the parser. There are preprocessors
+which allow you to alter the source before it is passed to the parser, 
+inline patterns which allow you to add, remove or override the syntax of
+any inline elements, and postprocessors which allow munging of the
+output of the parser before it is returned. If you really want to dive in, 
+there are also blockprocessors which are part of the core BlockParser.
+
+As the parser builds an [ElementTree][] object which is later rendered 
+as Unicode text, there are also some helpers provided to ease manipulation of 
+the tree. Each part of the API is discussed in its respective section below. 
+Additionaly, reading the source of some [[Available Extensions]] may be helpful.
+For example, the [[Footnotes]] extension uses most of the features documented 
+here.
+
+* [Preprocessors][]
+* [InlinePatterns][]
+* [Treeprocessors][] 
+* [Postprocessors][]
+* [BlockParser][]
+* [Working with the ElementTree][]
+* [Integrating your code into Markdown][]
+    * [extendMarkdown][]
+    * [OrderedDict][]
+    * [registerExtension][]
+    * [Config Settings][]
+    * [makeExtension][]
+
+<h3 id="preprocessors">Preprocessors</h3>
+
+Preprocessors munge the source text before it is passed into the Markdown 
+core. This is an excellent place to clean up bad syntax, extract things the 
+parser may otherwise choke on and perhaps even store it for later retrieval.
+
+Preprocessors should inherit from ``markdown.preprocessors.Preprocessor`` and 
+implement a ``run`` method with one argument ``lines``. The ``run`` method of 
+each Preprocessor will be passed the entire source text as a list of Unicode 
+strings. Each string will contain one line of text. The ``run`` method should 
+return either that list, or an altered list of Unicode strings.
+
+A pseudo example:
+
+    from markdown.preprocessors import Preprocessor
+
+    class MyPreprocessor(Preprocessor):
+        def run(self, lines):
+            new_lines = []
+            for line in lines:
+                m = MYREGEX.match(line)
+                if m:
+                    # do stuff
+                else:
+                    new_lines.append(line)
+            return new_lines
+
+<h3 id="inlinepatterns">Inline Patterns</h3>
+
+Inline Patterns implement the inline HTML element syntax for Markdown such as
+``*emphasis*`` or ``[links](http://example.com)``. Pattern objects should be 
+instances of classes that inherit from ``markdown.inlinepatterns.Pattern`` or 
+one of its children. Each pattern object uses a single regular expression and 
+must have the following methods:
+
+* **``getCompiledRegExp()``**: 
+
+    Returns a compiled regular expression.
+
+* **``handleMatch(m)``**: 
+
+    Accepts a match object and returns an ElementTree element of a plain 
+    Unicode string.
+
+Note that any regular expression returned by ``getCompiledRegExp`` must capture
+the whole block. Therefore, they should all start with ``r'^(.*?)'`` and end
+with ``r'(.*?)!'``. When using the default ``getCompiledRegExp()`` method 
+provided in the ``Pattern`` you can pass in a regular expression without that 
+and ``getCompiledRegExp`` will wrap your expression for you and set the 
+`re.DOTALL` and `re.UNICODE` flags. This means that the first group of your 
+match will be ``m.group(2)`` as ``m.group(1)`` will match everything before the
+pattern.
+
+For an example, consider this simplified emphasis pattern:
+
+    from markdown.inlinepatterns import Pattern
+    from markdown.util import etree
+
+    class EmphasisPattern(Pattern):
+        def handleMatch(self, m):
+            el = etree.Element('em')
+            el.text = m.group(3)
+            return el
+
+As discussed in [Integrating Your Code Into Markdown][], an instance of this
+class will need to be provided to Markdown. That instance would be created
+like so:
+
+    # an oversimplified regex
+    MYPATTERN = r'\*([^*]+)\*'
+    # pass in pattern and create instance
+    emphasis = EmphasisPattern(MYPATTERN)
+
+Actually it would not be necessary to create that pattern (and not just because
+a more sophisticated emphasis pattern already exists in Markdown). The fact is,
+that example pattern is not very DRY. A pattern for `**strong**` text would
+be almost identical, with the exception that it would create a 'strong' element.
+Therefore, Markdown provides a number of generic pattern classes that can 
+provide some common functionality. For example, both emphasis and strong are
+implemented with separate instances of the ``SimpleTagPettern`` listed below. 
+Feel free to use or extend any of the Pattern classes found at `markdown.inlinepatterns`.
+
+**Generic Pattern Classes**
+
+* **``SimpleTextPattern(pattern)``**:
+
+    Returns simple text of ``group(2)`` of a ``pattern``.
+
+* **``SimpleTagPattern(pattern, tag)``**:
+
+    Returns an element of type "`tag`" with a text attribute of ``group(3)``
+    of a ``pattern``. ``tag`` should be a string of a HTML element (i.e.: 'em').
+
+* **``SubstituteTagPattern(pattern, tag)``**:
+
+    Returns an element of type "`tag`" with no children or text (i.e.: 'br').
+
+There may be other Pattern classes in the Markdown source that you could extend
+or use as well. Read through the source and see if there is anything you can 
+use. You might even get a few ideas for different approaches to your specific
+situation.
+
+<h3 id="treeprocessors">Treeprocessors</h3>
+
+Treeprocessors manipulate an ElemenTree object after it has passed through the
+core BlockParser. This is where additional manipulation of the tree takes
+place. Additionally, the InlineProcessor is a Treeprocessor which steps through
+the tree and runs the InlinePatterns on the text of each Element in the tree.
+
+A Treeprocessor should inherit from ``markdown.treeprocessors.Treeprocessor``,
+over-ride the ``run`` method which takes one argument ``root`` (an Elementree 
+object) and returns either that root element or a modified root element.
+
+A pseudo example:
+
+    from markdown.treprocessors import Treeprocessor
+
+    class MyTreeprocessor(Treeprocessor):
+        def run(self, root):
+            #do stuff
+            return my_modified_root
+
+For specifics on manipulating the ElementTree, see 
+[Working with the ElementTree][] below.
+
+<h3 id="postprocessors">Postprocessors</h3>
+
+Postprocessors manipulate the document after the ElementTree has been 
+serialized into a string. Postprocessors should be used to work with the
+text just before output.
+
+A Postprocessor should inherit from ``markdown.postprocessors.Postprocessor`` 
+and over-ride the ``run`` method which takes one argument ``text`` and returns 
+a Unicode string.
+
+Postprocessors are run after the ElementTree has been serialized back into 
+Unicode text.  For example, this may be an appropriate place to add a table of 
+contents to a document:
+
+    from markdown.postprocessors import Postprocessor
+
+    class TocPostprocessor(Postprocessor):
+        def run(self, text):
+            return MYMARKERRE.sub(MyToc, text)
+
+<h3 id="blockparser">BlockParser</h3>
+
+Sometimes, pre/tree/postprocessors and Inline Patterns aren't going to do what 
+you need. Perhaps you want a new type of block type that needs to be integrated 
+into the core parsing. In such a situation, you can add/change/remove 
+functionality of the core ``BlockParser``. The BlockParser is composed of a
+number of Blockproccessors. The BlockParser steps through each block of text
+(split by blank lines) and passes each block to the appropriate Blockprocessor.
+That Blockprocessor parses the block and adds it to the ElementTree. The
+[[Definition Lists]] extension would be a good example of an extension that
+adds/modifies Blockprocessors.
+
+A Blockprocessor should inherit from ``markdown.blockprocessors.BlockProcessor``
+and implement both the ``test`` and ``run`` methods.
+
+The ``test`` method is used by BlockParser to identify the type of block.
+Therefore the ``test`` method must return a boolean value. If the test returns
+``True``, then the BlockParser will call that Blockprocessor's ``run`` method.
+If it returns ``False``, the BlockParser will move on to the next 
+BlockProcessor.
+
+The **``test``** method takes two arguments:
+
+* **``parent``**: The parent etree Element of the block. This can be useful as
+  the block may need to be treated differently if it is inside a list, for
+  example.
+
+* **``block``**: A string of the current block of text. The test may be a 
+  simple string method (such as ``block.startswith(some_text)``) or a complex 
+  regular expression.
+
+The **``run``** method takes two arguments:
+
+* **``parent``**: A pointer to the parent etree Element of the block. The run 
+  method will most likely attach additional nodes to this parent. Note that
+  nothing is returned by the method. The Elementree object is altered in place.
+
+* **``blocks``**: A list of all remaining blocks of the document. Your run 
+  method must remove (pop) the first block from the list (which it altered in
+  place - not returned) and parse that block. You may find that a block of text
+  legitimately contains multiple block types. Therefore, after processing the 
+  first type, your processor can insert the remaining text into the beginning
+  of the ``blocks`` list for future parsing.
+
+Please be aware that a single block can span multiple text blocks. For example,
+The official Markdown syntax rules state that a blank line does not end a
+Code Block. If the next block of text is also indented, then it is part of
+the previous block. Therefore, the BlockParser was specifically designed to 
+address these types of situations. If you notice the ``CodeBlockProcessor``,
+in the core, you will note that it checks the last child of the ``parent``.
+If the last child is a code block (``<pre><code>...</code></pre>``), then it
+appends that block to the previous code block rather than creating a new 
+code block.
+
+Each BlockProcessor has the following utility methods available:
+
+* **``lastChild(parent)``**: 
+
+    Returns the last child of the given etree Element or ``None`` if it had no 
+    children.
+
+* **``detab(text)``**: 
+
+    Removes one level of indent (four spaces by default) from the front of each
+    line of the given text string.
+
+* **``looseDetab(text, level)``**: 
+
+    Removes "level" levels of indent (defaults to 1) from the front of each line 
+    of the given text string. However, this methods allows secondary lines to 
+    not be indented as does some parts of the Markdown syntax.
+
+Each BlockProcessor also has a pointer to the containing BlockParser instance at
+``self.parser``, which can be used to check or alter the state of the parser.
+The BlockParser tracks it's state in a stack at ``parser.state``. The state
+stack is an instance of the ``State`` class.
+
+**``State``** is a subclass of ``list`` and has the additional methods:
+
+* **``set(state)``**: 
+
+    Set a new state to string ``state``. The new state is appended to the end 
+    of the stack.
+
+* **``reset()``**: 
+
+    Step back one step in the stack. The last state at the end is removed from 
+    the stack.
+
+* **``isstate(state)``**: 
+
+    Test that the top (current) level of the stack is of the given string 
+    ``state``.
+
+Note that to ensure that the state stack doesn't become corrupted, each time a
+state is set for a block, that state *must* be reset when the parser finishes
+parsing that block.
+
+An instance of the **``BlockParser``** is found at ``Markdown.parser``.
+``BlockParser`` has the following methods:
+
+* **``parseDocument(lines)``**: 
+
+    Given a list of lines, an ElementTree object is returned. This should be 
+    passed an entire document and is the only method the ``Markdown`` class 
+    calls directly.
+
+* **``parseChunk(parent, text)``**: 
+
+    Parses a chunk of markdown text composed of multiple blocks and attaches 
+    those blocks to the ``parent`` Element. The ``parent`` is altered in place 
+    and nothing is returned. Extensions would most likely use this method for 
+    block parsing.
+
+* **``parseBlocks(parent, blocks)``**: 
+
+    Parses a list of blocks of text and attaches those blocks to the ``parent``
+    Element. The ``parent`` is altered in place and nothing is returned. This 
+    method will generally only be used internally to recursively parse nested 
+    blocks of text.
+
+While is is not recommended, an extension could subclass or completely replace
+the ``BlockParser``. The new class would have to provide the same public API.
+However, be aware that other extensions may expect the core parser provided
+and will not work with such a drastically different parser.
+
+<h3 id="working_with_et">Working with the ElementTree</h3>
+
+As mentioned, the Markdown parser converts a source document to an 
+[ElementTree][] object before serializing that back to Unicode text. 
+Markdown has provided some helpers to ease that manipulation within the context 
+of the Markdown module.
+
+First, to get access to the ElementTree module import ElementTree from 
+``markdown`` rather than importing it directly. This will ensure you are using 
+the same version of ElementTree as markdown. The module is found at 
+``markdown.util.etree`` within Markdown.
+
+    from markdown.util import etree
+    
+``markdown.util.etree`` tries to import ElementTree from any known location, 
+first as a standard library module (from ``xml.etree`` in Python 2.5), then as 
+a third party package (``Elementree``). In each instance, ``cElementTree`` is 
+tried first, then ``ElementTree`` if the faster C implementation is not 
+available on your system.
+
+Sometimes you may want text inserted into an element to be parsed by 
+[InlinePatterns][]. In such a situation, simply insert the text as you normally
+would and the text will be automatically run through the InlinePatterns. 
+However, if you do *not* want some text to be parsed by InlinePatterns,
+then insert the text as an ``AtomicString``.
+
+    from markdown.util import AtomicString
+    some_element.text = AtomicString(some_text)
+
+Here's a basic example which creates an HTML table (note that the contents of 
+the second cell (``td2``) will be run through InlinePatterns latter):
+
+    table = etree.Element("table") 
+    table.set("cellpadding", "2")                      # Set cellpadding to 2
+    tr = etree.SubElement(table, "tr")                 # Add child tr to table
+    td1 = etree.SubElement(tr, "td")                   # Add child td1 to tr
+    td1.text = markdown.AtomicString("Cell content")   # Add plain text content
+    td2 = etree.SubElement(tr, "td")                   # Add second td to tr
+    td2.text = "*text* with **inline** formatting."    # Add markup text
+    table.tail = "Text after table"                    # Add text after table
+
+You can also manipulate an existing tree. Consider the following example which 
+adds a ``class`` attribute to ``<a>`` elements:
+
+	def set_link_class(self, element):
+		for child in element: 
+		    if child.tag == "a":
+                child.set("class", "myclass") #set the class attribute
+            set_link_class(child) # run recursively on children
+
+For more information about working with ElementTree see the ElementTree
+[Documentation](http://effbot.org/zone/element-index.htm) 
+([Python Docs](http://docs.python.org/lib/module-xml.etree.ElementTree.html)).
+
+<h3 id="integrating_into_markdown">Integrating Your Code Into Markdown</h3>
+
+Once you have the various pieces of your extension built, you need to tell 
+Markdown about them and ensure that they are run in the proper sequence. 
+Markdown accepts a ``Extension`` instance for each extension. Therefore, you
+will need to define a class that extends ``markdown.extensions.Extension`` and 
+over-rides the ``extendMarkdown`` method. Within this class you will manage 
+configuration options for your extension and attach the various processors and 
+patterns to the Markdown instance. 
+
+It is important to note that the order of the various processors and patterns 
+matters. For example, if we replace ``http://...`` links with <a> elements, and 
+*then* try to deal with  inline html, we will end up with a mess. Therefore, 
+the various types of processors and patterns are stored within an instance of 
+the Markdown class in [OrderedDict][]s. Your ``Extension`` class will need to 
+manipulate those OrderedDicts appropriately. You may insert instances of your 
+processors and patterns into the appropriate location in an OrderedDict, remove
+a built-in instance, or replace a built-in instance with your own.
+
+<h4 id="extendmarkdown">extendMarkdown</h4>
+
+The ``extendMarkdown`` method of a ``markdown.extensions.Extension`` class 
+accepts two arguments:
+
+* **``md``**:
+
+    A pointer to the instance of the Markdown class. You should use this to 
+    access the [OrderedDict][]s of processors and patterns. They are found 
+    under the following attributes:
+
+    * ``md.preprocessors``
+    * ``md.inlinePatterns``
+    * ``md.parser.blockprocessors``
+    * ``md.treepreprocessors``
+    * ``md.postprocessors``
+
+    Some other things you may want to access in the markdown instance are:
+
+    * ``md.htmlStash``
+    * ``md.output_formats``
+    * ``md.set_output_format()``
+    * ``md.registerExtension()``
+    * ``md.html_replacement_text``
+    * ``md.tab_length``
+    * ``md.enable_attributes``
+    * ``md.smart_emphasis``
+
+* **``md_globals``**:
+
+    Contains all the various global variables within the markdown module.
+
+Of course, with access to those items, theoretically you have the option to 
+changing anything through various [monkey_patching][] techniques. However, you 
+should be aware that the various undocumented or private parts of markdown 
+may change without notice and your monkey_patches may break with a new release.
+Therefore, what you really should be doing is inserting processors and patterns
+into the markdown pipeline. Consider yourself warned.
+
+[monkey_patching]: http://en.wikipedia.org/wiki/Monkey_patch
+
+A simple example:
+
+    from markdown.extensions import Extension
+
+    class MyExtension(Extension):
+        def extendMarkdown(self, md, md_globals):
+            # Insert instance of 'mypattern' before 'references' pattern
+            md.inlinePatterns.add('mypattern', MyPattern(md), '<references')
+
+<h4 id="ordereddict">OrderedDict</h4>
+
+An OrderedDict is a dictionary like object that retains the order of it's
+items. The items are ordered in the order in which they were appended to
+the OrderedDict. However, an item can also be inserted into the OrderedDict
+in a specific location in relation to the existing items.
+
+Think of OrderedDict as a combination of a list and a dictionary as it has 
+methods common to both. For example, you can get and set items using the 
+``od[key] = value`` syntax and the methods ``keys()``, ``values()``, and 
+``items()`` work as expected with the keys, values and items returned in the 
+proper order. At the same time, you can use ``insert()``, ``append()``, and 
+``index()`` as you would with a list.
+
+Generally speaking, within Markdown extensions you will be using the special 
+helper method ``add()`` to add additional items to an existing OrderedDict. 
+
+The ``add()`` method accepts three arguments:
+
+* **``key``**: A string. The key is used for later reference to the item.
+
+* **``value``**: The object instance stored in this item.
+
+* **``location``**: Optional. The items location in relation to other items. 
+
+    Note that the location can consist of a few different values:
+
+    * The special strings ``"_begin"`` and ``"_end"`` insert that item at the 
+      beginning or end of the OrderedDict respectively. 
+    
+    * A less-than sign (``<``) followed by an existing key (i.e.: 
+      ``"<somekey"``) inserts that item before the existing key.
+    
+    * A greater-than sign (``>``) followed by an existing key (i.e.: 
+      ``">somekey"``) inserts that item after the existing key. 
+
+Consider the following example:
+
+    >>> from markdown.odict import OrderedDict
+    >>> od = OrderedDict()
+    >>> od['one'] =  1           # The same as: od.add('one', 1, '_begin')
+    >>> od['three'] = 3          # The same as: od.add('three', 3, '>one')
+    >>> od['four'] = 4           # The same as: od.add('four', 4, '_end')
+    >>> od.items()
+    [("one", 1), ("three", 3), ("four", 4)]
+
+Note that when building an OrderedDict in order, the extra features of the
+``add`` method offer no real value and are not necessary. However, when 
+manipulating an existing OrderedDict, ``add`` can be very helpful. So let's 
+insert another item into the OrderedDict.
+
+    >>> od.add('two', 2, '>one')         # Insert after 'one'
+    >>> od.values()
+    [1, 2, 3, 4]
+
+Now let's insert another item.
+
+    >>> od.add('twohalf', 2.5, '<three') # Insert before 'three'
+    >>> od.keys()
+    ["one", "two", "twohalf", "three", "four"]
+
+Note that we also could have set the location of "twohalf" to be 'after two'
+(i.e.: ``'>two'``). However, it's unlikely that you will have control over the 
+order in which extensions will be loaded, and this could affect the final 
+sorted order of an OrderedDict. For example, suppose an extension adding 
+'twohalf' in the above examples was loaded before a separate  extension which 
+adds 'two'. You may need to take this into consideration when adding your 
+extension components to the various markdown OrderedDicts.
+
+Once an OrderedDict is created, the items are available via key:
+
+    MyNode = od['somekey']
+
+Therefore, to delete an existing item:
+
+    del od['somekey']
+
+To change the value of an existing item (leaving location unchanged):
+
+    od['somekey'] = MyNewObject()
+
+To change the location of an existing item:
+
+    t.link('somekey', '<otherkey')
+
+<h4 id="registerextension">registerExtension</h4>
+
+Some extensions may need to have their state reset between multiple runs of the
+Markdown class. For example, consider the following use of the [[Footnotes]] 
+extension:
+
+    md = markdown.Markdown(extensions=['footnotes'])
+    html1 = md.convert(text_with_footnote)
+    md.reset()
+    html2 = md.convert(text_without_footnote)
+
+Without calling ``reset``, the footnote definitions from the first document will
+be inserted into the second document as they are still stored within the class
+instance. Therefore the ``Extension`` class needs to define a ``reset`` method
+that will reset the state of the extension (i.e.: ``self.footnotes = {}``).
+However, as many extensions do not have a need for ``reset``, ``reset`` is only
+called on extensions that are registered.
+
+To register an extension, call ``md.registerExtension`` from within your 
+``extendMarkdown`` method:
+
+
+    def extendMarkdown(self, md, md_globals):
+        md.registerExtension(self)
+        # insert processors and patterns here
+
+Then, each time ``reset`` is called on the Markdown instance, the ``reset`` 
+method of each registered extension will be called as well. You should also
+note that ``reset`` will be called on each registered extension after it is
+initialized the first time. Keep that in mind when over-riding the extension's
+``reset`` method.
+
+<h4 id="configsettings">Config Settings</h4>
+
+If an extension uses any parameters that the user may want to change,
+those parameters should be stored in ``self.config`` of your 
+``markdown.Extension`` class in the following format:
+
+    self.config = {parameter_1_name : [value1, description1],
+                   parameter_2_name : [value2, description2] }
+
+When stored this way the config parameters can be over-ridden from the
+command line or at the time Markdown is initiated:
+
+    markdown.py -x myextension(SOME_PARAM=2) inputfile.txt > output.txt
+
+Note that parameters should always be assumed to be set to string
+values, and should be converted at run time. For example:
+
+    i = int(self.getConfig("SOME_PARAM"))
+
+<h4 id="makeextension">makeExtension</h4>
+
+Each extension should ideally be placed in its own module starting
+with the  ``mdx_`` prefix (e.g. ``mdx_footnotes.py``).  The module must
+provide a module-level function called ``makeExtension`` that takes
+an optional parameter consisting of a dictionary of configuration over-rides 
+and returns an instance of the extension.  An example from the footnote 
+extension:
+
+    def makeExtension(configs=None) :
+        return FootnoteExtension(configs=configs)
+
+By following the above example, when Markdown is passed the name of your 
+extension as a string (i.e.: ``'footnotes'``), it will automatically import
+the module and call the ``makeExtension`` function initiating your extension.
+
+You may have noted that the extensions packaged with Python-Markdown do not
+use the ``mdx_`` prefix in their module names. This is because they are all
+part of the ``markdown.extensions`` package. Markdown will first try to import
+from ``markdown.extensions.extname`` and upon failure, ``mdx_extname``. If both
+fail, Markdown will continue without the extension.
+
+However, Markdown will also accept an already existing instance of an extension.
+For example:
+
+    import markdown
+    import myextension
+    configs = {...}
+    myext = myextension.MyExtension(configs=configs)
+    md = markdown.Markdown(extensions=[myext])
+
+This is useful if you need to implement a large number of extensions with more
+than one residing in a module.
+
+[Preprocessors]: #preprocessors
+[InlinePatterns]: #inlinepatterns
+[Treeprocessors]: #treeprocessors
+[Postprocessors]: #postprocessors
+[BlockParser]: #blockparser
+[Working with the ElementTree]: #working_with_et
+[Integrating your code into Markdown]: #integrating_into_markdown
+[extendMarkdown]: #extendmarkdown
+[OrderedDict]: #ordereddict
+[registerExtension]: #registerextension
+[Config Settings]: #configsettings
+[makeExtension]: #makeextension
+[ElementTree]: http://effbot.org/zone/element-index.htm