annotate emeraldtree/HTMLTreeBuilder.py @ 51:847897e5fab8

HTMLTreeBuilder - Make namespace aware, add testcases
author Bastian Blank <bblank@thinkmo.de>
date Sun, 21 Sep 2008 11:01:19 +0200
parents e647f30cc08e
children
rev   line source
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
1 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
2 # ElementTree
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
3 # $Id: HTMLTreeBuilder.py 3265 2007-09-06 20:42:00Z fredrik $
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
4 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
5 # a simple tree builder, for HTML input
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
6 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
7 # history:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
8 # 2002-04-06 fl created
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
9 # 2002-04-07 fl ignore IMG and HR end tags
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
10 # 2002-04-07 fl added support for 1.5.2 and later
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
11 # 2003-04-13 fl added HTMLTreeBuilder alias
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
12 # 2004-12-02 fl don't feed non-ASCII charrefs/entities as 8-bit strings
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
13 # 2004-12-05 fl don't feed non-ASCII CDATA as 8-bit strings
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
14 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
15 # Copyright (c) 1999-2004 by Fredrik Lundh. All rights reserved.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
16 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
17 # fredrik@pythonware.com
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
18 # http://www.pythonware.com
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
19 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
20 # --------------------------------------------------------------------
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
21 # The ElementTree toolkit is
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
22 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
23 # Copyright (c) 1999-2007 by Fredrik Lundh
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
24 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
25 # By obtaining, using, and/or copying this software and/or its
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
26 # associated documentation, you agree that you have read, understood,
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
27 # and will comply with the following terms and conditions:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
28 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
29 # Permission to use, copy, modify, and distribute this software and
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
30 # its associated documentation for any purpose and without fee is
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
31 # hereby granted, provided that the above copyright notice appears in
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
32 # all copies, and that both that copyright notice and this permission
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
33 # notice appear in supporting documentation, and that the name of
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
34 # Secret Labs AB or the author not be used in advertising or publicity
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
35 # pertaining to distribution of the software without specific, written
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
36 # prior permission.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
37 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
38 # SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
39 # TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT-
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
40 # ABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
41 # BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
42 # DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
43 # WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
44 # ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
45 # OF THIS SOFTWARE.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
46 # --------------------------------------------------------------------
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
47
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
48 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
49 # Tools to build element trees from HTML files.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
50 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
51
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
52 import htmlentitydefs
47
e647f30cc08e remove backwards compat code for python < 2.4
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 1
diff changeset
53 import re
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
54 import mimetools, StringIO
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
55 from HTMLParser import HTMLParser as HTMLParserBase
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
56
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
57 import ElementTree
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
58
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
59
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
60 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
61 # ElementTree builder for HTML source code. This builder converts an
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
62 # HTML document or fragment to an ElementTree.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
63 # <p>
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
64 # The parser is relatively picky, and requires balanced tags for most
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
65 # elements. However, elements belonging to the following group are
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
66 # automatically closed: P, LI, TR, TH, and TD. In addition, the
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
67 # parser automatically inserts end tags immediately after the start
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
68 # tag, and ignores any end tags for the following group: IMG, HR,
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
69 # META, and LINK.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
70 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
71 # @keyparam builder Optional builder object. If omitted, the parser
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
72 # uses the standard <b>elementtree</b> builder.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
73 # @keyparam encoding Optional character encoding, if known. If omitted,
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
74 # the parser looks for META tags inside the document. If no tags
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
75 # are found, the parser defaults to ISO-8859-1. Note that if your
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
76 # document uses a non-ASCII compatible encoding, you must decode
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
77 # the document before parsing.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
78 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
79 # @see elementtree.ElementTree
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
80
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
81 class HTMLParser(HTMLParserBase):
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
82 AUTOCLOSE = "p", "li", "tr", "th", "td", "head", "body"
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
83 IGNOREEND = "img", "hr", "meta", "link", "br", "input", "col"
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
84
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
85 namespace = "http://www.w3.org/1999/xhtml"
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
86
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
87 def __init__(self, builder=None, encoding=None):
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
88 self.__stack = []
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
89 if builder is None:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
90 builder = ElementTree.TreeBuilder()
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
91 self.__builder = builder
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
92 self.encoding = encoding or "iso-8859-1"
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
93 HTMLParserBase.__init__(self)
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
94
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
95 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
96 # Flushes parser buffers, and return the root element.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
97 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
98 # @return An Element instance.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
99
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
100 def close(self):
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
101 HTMLParserBase.close(self)
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
102 return self.__builder.close()
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
103
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
104 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
105 # (Internal) Handles start tags.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
106
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
107 def handle_starttag(self, tag, attrs):
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
108 tag = ElementTree.QName(tag.lower(), self.namespace)
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
109 if tag.name == "meta":
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
110 # look for encoding directives
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
111 http_equiv = content = None
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
112 for k, v in attrs:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
113 if k == "http-equiv":
47
e647f30cc08e remove backwards compat code for python < 2.4
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 1
diff changeset
114 http_equiv = v.lower()
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
115 elif k == "content":
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
116 content = v
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
117 if http_equiv == "content-type" and content:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
118 # use mimetools to parse the http header
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
119 header = mimetools.Message(
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
120 StringIO.StringIO("%s: %s\n\n" % (http_equiv, content))
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
121 )
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
122 encoding = header.getparam("charset")
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
123 if encoding:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
124 self.encoding = encoding
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
125 if tag.name in self.AUTOCLOSE:
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
126 if self.__stack and self.__stack[-1] == tag:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
127 self.handle_endtag(tag)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
128 self.__stack.append(tag)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
129 attrib = {}
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
130 if attrs:
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
131 for key, value in attrs:
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
132 # Handle short attributes
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
133 if value is None:
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
134 value = key
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
135 key = ElementTree.QName(key.lower(), self.namespace)
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
136 attrib[key] = value
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
137 self.__builder.start(tag, attrib)
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
138 if tag.name in self.IGNOREEND:
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
139 self.__stack.pop()
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
140 self.__builder.end(tag)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
141
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
142 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
143 # (Internal) Handles end tags.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
144
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
145 def handle_endtag(self, tag):
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
146 if not isinstance(tag, ElementTree.QName):
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
147 tag = ElementTree.QName(tag.lower(), self.namespace)
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
148 if tag.name in self.IGNOREEND:
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
149 return
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
150 lasttag = self.__stack.pop()
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
151 if tag != lasttag and lasttag.name in self.AUTOCLOSE:
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
152 self.handle_endtag(lasttag)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
153 self.__builder.end(tag)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
154
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
155 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
156 # (Internal) Handles character references.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
157
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
158 def handle_charref(self, char):
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
159 if char[:1] == "x":
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
160 char = int(char[1:], 16)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
161 else:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
162 char = int(char)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
163 if 0 <= char < 128:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
164 self.__builder.data(chr(char))
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
165 else:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
166 self.__builder.data(unichr(char))
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
167
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
168 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
169 # (Internal) Handles entity references.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
170
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
171 def handle_entityref(self, name):
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
172 entity = htmlentitydefs.entitydefs.get(name)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
173 if entity:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
174 if len(entity) == 1:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
175 entity = ord(entity)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
176 else:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
177 entity = int(entity[2:-1])
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
178 if 0 <= entity < 128:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
179 self.__builder.data(chr(entity))
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
180 else:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
181 self.__builder.data(unichr(entity))
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
182 else:
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
183 self.unknown_entityref(name)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
184
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
185 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
186 # (Internal) Handles character data.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
187
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
188 def handle_data(self, data):
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
189 if isinstance(data, str):
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
190 # convert to unicode, but only if necessary
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
191 data = unicode(data, self.encoding, "ignore")
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
192 self.__builder.data(data)
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
193
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
194 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
195 # (Hook) Handles unknown entity references. The default action
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
196 # is to ignore unknown entities.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
197
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
198 def unknown_entityref(self, name):
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
199 pass # ignore by default; override if necessary
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
200
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
201 ##
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
202 # An alias for the <b>HTMLParser</b> class.
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
203
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
204 TreeBuilder = HTMLTreeBuilder = HTMLParser
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
205
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
206 ##
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
207 # Parse an HTML document or document fragment.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
208 #
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
209 # @param source A filename or file object containing HTML data.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
210 # @param encoding Optional character encoding, if known. If omitted,
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
211 # the parser looks for META tags inside the document. If no tags
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
212 # are found, the parser defaults to ISO-8859-1.
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
213 # @return An ElementTree instance
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
214
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
215 def parse(source, encoding=None):
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
216 return ElementTree.parse(source, HTMLTreeBuilder(encoding=encoding))
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
217
51
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
218 def HTML(text):
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
219 parser = HTMLParser()
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
220 parser.feed(text)
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
221 return parser.close()
847897e5fab8 HTMLTreeBuilder - Make namespace aware, add testcases
Bastian Blank <bblank@thinkmo.de>
parents: 47
diff changeset
222
0
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
223 if __name__ == "__main__":
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
224 import sys
5169fce2d144 Import ElementTree (1.3a3-20070912-preview).
Bastian Blank <bblank@thinkmo.de>
parents:
diff changeset
225 ElementTree.dump(parse(open(sys.argv[1])))