Support custom labels in TOC. (#700)

New `toc_tokens` attribute on Markdown class.
Contains the raw tokens used to build the Table of Contents. Users can 
use this to build their own custom Table of Contents rather than needing 
to parse the HTML available on the `toc` attribute of the Markdown 
class.

4 files changed, 108 insertions, 12 deletions

diff --git a/docs/change_log/release-3.0.md b/docs/change_log/release-3.0.md
index 85a5d27..ab6b83e 100644
--- a/docs/change_log/release-3.0.md
+++ b/docs/change_log/release-3.0.md
@@ -202,5 +202,17 @@ The following new features have been included in the release:
 * A new `toc_depth` parameter has been added to the
   [Table of Contents Extension](../extensions/toc.md).
 
+* A new `toc_tokens` attribute has been added to the Markdown class by the
+  [Table of Contents Extension](../extensions/toc.md), which contains the raw
+  tokens used to build the Table of Contents. Users can use this to build their
+  own custom Table of Contents rather than needing to parse the HTML available
+  on the `toc` attribute of the Markdown class.
+
+* When the [Table of Contents Extension](../extensions/toc.md) is used in
+  conjunction with the [Attribute Lists Extension](../extensions/attr_list.md)
+  and a `data-toc-label` attribute is defined on a header, the content of the
+  `data-toc-label` attribute is now used as the content of the Table of Contents
+  item for that header.
+
 * Additional CSS class names can be appended to
   [Admonitions](../extensions/admonition.md).
diff --git a/docs/extensions/toc.md b/docs/extensions/toc.md
index 5038428..f6111b2 100644
--- a/docs/extensions/toc.md
+++ b/docs/extensions/toc.md
@@ -56,7 +56,7 @@ would generate the following output:
   </ul>
 </div>
 <h1 id="header-1">Header 1</h1>
-<h1 id="header-2">Header 2</h1>
+<h2 id="header-2">Header 2</h2>
 ```
 
 Regardless of whether a `marker` is found in the document (or disabled), the
@@ -70,6 +70,67 @@ template. For example:
 >>> page = render_some_template(context={'body': html, 'toc': md.toc})
 ```
 
+The `toc_tokens` attribute is also available on the Markdown class and contains
+a nested list of dict objects. For example, the above document would result in
+the following object at `md.toc_tokens`:
+
+```python
+[
+    {
+        'level': 1,
+        'id': 'header-1',
+        'name': 'Header 1',
+        'children': [
+            {'level': 2, 'id': 'header-2', 'name': 'Header 2', 'children':[]}
+        ]
+    }
+]
+```
+
+Note that the `level` refers to the `hn` level. In other words, `<h1>` is level
+`1` and `<h2>` is level `2`, etc. Be aware that improperly nested levels in the
+input may result in odd nesting of the output.
+
+### Custom Labels
+
+In most cases, the text label in the Table of Contents should match the text of
+the header. However, occasionally that is not desirable. In that case, if this
+extension is used in conjunction with the [Attribute Lists Extension] and a
+`data-toc-label` attribute is defined on the header, then the contents of that
+attribute will be used as the text label for the item in the Table of Contents.
+For example, the following Markdown:
+
+[Attribute Lists Extension]: attr_list.md
+
+```md
+[TOC]
+
+# Functions
+
+## `markdown.markdown(text [, **kwargs])` { #markdown data-toc-label='markdown.markdown' }
+```
+would generate the following output:
+
+```html
+<div class="toc">
+  <ul>
+    <li><a href="#functions">Functions</a></li>
+      <ul>
+        <li><a href="#markdown">markdown.markdown</a></li>
+      </ul>
+  </ul>
+</div>
+<h1 id="functions">Functions</h1>
+<h2 id="markdown"><code>markdown.markdown(text [, **kwargs])</code></h2>
+```
+
+Notice that the text in the Table of Contents is much cleaner and easier to read
+in the context of a Table of Contents. The `data-toc-label` is not included in
+the HTML header element. Also note that the ID was manually defined in the
+attribute list to provide a cleaner URL when linking to the header. If the ID is
+not manually defined, it is always derived from the text of the header, never
+from the `data-toc-label` attribute.
+
 Usage
 -----
 
diff --git a/markdown/extensions/toc.py b/markdown/extensions/toc.py
index 2741037..f6121c2 100644
--- a/markdown/extensions/toc.py
+++ b/markdown/extensions/toc.py
@@ -247,15 +247,20 @@ class TocTreeprocessor(Treeprocessor):
                 toc_tokens.append({
                     'level': int(el.tag[-1]),
                     'id': el.attrib["id"],
-                    'name': text
+                    'name': el.attrib.get('data-toc-label', text)
                 })
 
+                # Remove the data-toc-label attribute as it is no longer needed
+                if 'data-toc-label' in el.attrib:
+                    del el.attrib['data-toc-label']
+
                 if self.use_anchors:
                     self.add_anchor(el, el.attrib["id"])
                 if self.use_permalinks:
                     self.add_permalink(el, el.attrib["id"])
 
-        div = self.build_toc_div(nest_toc_tokens(toc_tokens))
+        toc_tokens = nest_toc_tokens(toc_tokens)
+        div = self.build_toc_div(toc_tokens)
         if self.marker:
             self.replace_marker(doc, div)
 
@@ -263,6 +268,7 @@ class TocTreeprocessor(Treeprocessor):
         toc = self.md.serializer(div)
         for pp in self.md.postprocessors:
             toc = pp.run(toc)
+        self.md.toc_tokens = toc_tokens
         self.md.toc = toc
 
 
@@ -310,6 +316,7 @@ class TocExtension(Extension):
 
     def reset(self):
         self.md.toc = ''
+        self.md.toc_tokens = []
 
 
 def makeExtension(**kwargs):  # pragma: no cover
diff --git a/tests/test_extensions.py b/tests/test_extensions.py
index 532ed6f..5489887 100644
--- a/tests/test_extensions.py
+++ b/tests/test_extensions.py
@@ -852,24 +852,40 @@ class TestTOC(TestCaseWithAssertStartsWith):
     def testWithAttrList(self):
         """ Test TOC with attr_list Extension. """
         md = markdown.Markdown(extensions=['toc', 'attr_list'])
-        text = '# Header 1\n\n## Header 2 { #foo }'
+        text = '# Header 1\n\n## Header 2 { #foo }\n\n## Header 3 { data-toc-label="Foo Bar"}'
         self.assertEqual(
             md.convert(text),
             '<h1 id="header-1">Header 1</h1>\n'
-            '<h2 id="foo">Header 2</h2>'
+            '<h2 id="foo">Header 2</h2>\n'
+            '<h2 id="header-3">Header 3</h2>'
         )
         self.assertEqual(
             md.toc,
             '<div class="toc">\n'
-              '<ul>\n'                                        # noqa
-                '<li><a href="#header-1">Header 1</a>'        # noqa
-                  '<ul>\n'                                    # noqa
-                    '<li><a href="#foo">Header 2</a></li>\n'  # noqa
-                  '</ul>\n'                                   # noqa
-                '</li>\n'                                     # noqa
-              '</ul>\n'                                       # noqa
+              '<ul>\n'                                           # noqa
+                '<li><a href="#header-1">Header 1</a>'           # noqa
+                  '<ul>\n'                                       # noqa
+                    '<li><a href="#foo">Header 2</a></li>\n'     # noqa
+                    '<li><a href="#header-3">Foo Bar</a></li>\n' # noqa
+                  '</ul>\n'                                      # noqa
+                '</li>\n'                                        # noqa
+              '</ul>\n'                                          # noqa
             '</div>\n'
         )
+        self.assertEqual(
+            md.toc_tokens,
+            [
+                {
+                    'level': 1,
+                    'id': 'header-1',
+                    'name': 'Header 1',
+                    'children': [
+                        {'level': 2, 'id': 'foo', 'name': 'Header 2', 'children': []},
+                        {'level': 2, 'id': 'header-3', 'name': 'Foo Bar', 'children': []}
+                    ]
+                }
+            ]
+        )
 
     def testUniqueFunc(self):
         """ Test 'unique' function. """