annotate MoinMoin/parser/text_csv.py @ 3641:00af8e564f96 SOC2008-START

parser.text_csv: removes empty lines in front of the csv table
author Reimar Bauer <rb.proj AT googlemail DOT com>
date Sun, 25 May 2008 00:43:25 +0200
parents 75db5e7c09e2
children 8fbdd3667488
rev   line source
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
1 # -*- coding: utf-8 -*-
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
2 """
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
3 MoinMoin - Parser for CSV data
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
4
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
5 This parser uses the databrowser widget to display the data.
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
6
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
7 It supports the following parser arguments:
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
8
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
9 * delimiter/separator: the delimiter to use instead of ;
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
10 * quotechar: quoting character, default off, must be ascii!
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
11 * show: comma-separated list of columns to show only
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
12 * hide: comma-separated list of columns to hide
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
13 * autofilter: comma-separated list of columns to equip with
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
14 auto-filter drop down
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
15 * name: name of the dataset
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
16 * link: comma separated list of columns that take links, separate
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
17 the link and the description with a space
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
18 * static_cols: comma-separated list of columns that are static
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
19 and present in each row
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
20 * static_vals: comma-separated list of values for those static
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
21 columns
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
22
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
23 The static column feature is only really useful if the dataset
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
24 postprocessed by some other plugin collecting data from multiple
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
25 wiki pages.
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
26
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
27 @copyright: 2007, 2008 Johannes Berg <johannes@sipsolutions.net>
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
28 @license: GNU GPL, see COPYING for details.
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
29 """
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
30
3340
afef9c8b2ed2 parser.text_csv: added Sniffer for delimiter
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3314
diff changeset
31 from csv import reader, QUOTE_NONE, QUOTE_MINIMAL, Sniffer
3635
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
32 from _csv import Error
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
33
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
34 from MoinMoin.util.dataset import TupleDataset, Column
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
35 from MoinMoin.widget.browser import DataBrowserWidget
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
36 from MoinMoin.wikiutil import escape
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
37
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
38
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
39 Dependencies = ['time']
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
40
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
41 class Parser:
3312
25c223e1b6e4 change extension back to .csv (as was the case with the old csv parser)
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3311
diff changeset
42 extensions = ['.csv']
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
43 Dependencies = []
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
44
3311
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
45 def _read_rows(self, r):
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
46 if self._first_row is not None:
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
47 yield self._first_row
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
48 for row in r:
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
49 yield row
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
50
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
51 def __init__(self, raw, request, **kw):
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
52 self.request = request
3311
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
53 self._first_row = None
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
54 formatter = request.formatter
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
55
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
56 # workaround csv.reader deficiency by encoding to utf-8
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
57 data = raw.encode('utf-8').split('\n')
3641
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
58
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
59 # workaraound to remove empty lines in front of the csv table
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
60 # if this is not done this parser shows only an empty line
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
61 count = 0
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
62 for txt in data:
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
63 if not txt:
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
64 count += 1
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
65 else:
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
66 data = data[count:]
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
67 break
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
68
3448
f26af7360865 parser.text_csv: fixed bug sniffing empty line
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3340
diff changeset
69 delimiter = ';'
3635
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
70 # Previous versions of this parser have used only the delimiter ";" (by default).
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
71 # This version now tries to sniff the delimiter from the list preferred_delimiters
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
72 # Although the Python csv sniffer had quite some changes from py 2.3 to 2.5.1, we try
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
73 # to avoid problems for the case it does not find a delimiter in some given data.
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
74 # Newer versions of the sniffer do raise an _csv.Error while older versions do
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
75 # return a whitespace as delimiter.
3448
f26af7360865 parser.text_csv: fixed bug sniffing empty line
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3340
diff changeset
76 if data[0]:
3635
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
77 try:
3641
00af8e564f96 parser.text_csv: removes empty lines in front of the csv table
Reimar Bauer <rb.proj AT googlemail DOT com>
parents: 3635
diff changeset
78 preferred_delimiters = [',', '\t', ';', ' ', ':']
3635
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
79 delimiter = Sniffer().sniff(data[0], preferred_delimiters).delimiter or ';'
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
80 except Error:
75db5e7c09e2 csv parser: fix delimiter sniffing troubles, fix tests
Thomas Waldmann <tw AT waldmann-edv DOT de>
parents: 3448
diff changeset
81 pass
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
82
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
83 visible = None
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
84 hiddenindexes = []
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
85 hiddencols = []
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
86 autofiltercols = []
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
87 staticcols = []
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
88 staticvals = []
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
89 linkcols = []
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
90 quotechar = '\x00' # can't be entered
3314
55841be09a2f csv reader: properly set quoting keyword
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3312
diff changeset
91 quoting = QUOTE_NONE
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
92 name = None
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
93 hdr = reader([kw.get('format_args', '').strip().encode('utf-8')], delimiter=" ")
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
94 args = hdr.next()
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
95
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
96 for arg in args:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
97 arg = arg.decode('utf-8')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
98 try:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
99 key, val = arg.split('=', 1)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
100 except:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
101 # handle compatibility with original 'csv' parser
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
102 if arg.startswith('-'):
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
103 try:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
104 hiddenindexes.append(int(arg[1:]) - 1)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
105 except ValueError:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
106 pass
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
107 else:
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
108 delimiter = arg.encode('utf-8')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
109 continue
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
110 if key == 'separator' or key == 'delimiter':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
111 delimiter = val.encode('utf-8')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
112 if key == 'quotechar':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
113 if val == val.encode('utf-8'):
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
114 quotechar = val.encode('utf-8')
3314
55841be09a2f csv reader: properly set quoting keyword
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3312
diff changeset
115 quoting = QUOTE_MINIMAL
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
116 elif key == 'show':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
117 visible = val.split(',')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
118 elif key == 'hide':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
119 hiddencols = val.split(',')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
120 elif key == 'autofilter':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
121 autofiltercols = val.split(',')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
122 elif key == 'name':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
123 name = val
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
124 elif key == 'static_cols':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
125 staticcols = val.split(',')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
126 elif key == 'static_vals':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
127 staticvals = val.split(',')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
128 elif key == 'link':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
129 linkcols = val.split(',')
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
130
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
131 if len(staticcols) > len(staticvals):
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
132 staticvals.extend([''] * (len(staticcols)-len(staticvals)))
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
133 elif len(staticcols) < len(staticvals):
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
134 staticvals = staticvals[:len(staticcols)]
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
135
3314
55841be09a2f csv reader: properly set quoting keyword
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3312
diff changeset
136 r = reader(data, delimiter=delimiter, quotechar=quotechar, quoting=quoting)
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
137 cols = map(lambda x: x.decode('utf-8'), r.next()) + staticcols
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
138
3311
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
139 self._show_header = True
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
140
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
141 if cols == staticcols:
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
142 try:
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
143 self._first_row = map(lambda x: x.decode('utf-8'), r.next())
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
144 cols = [None] * len(self._first_row) + staticcols
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
145 self._show_header = False
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
146 except StopIteration:
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
147 pass
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
148
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
149 num_entry_cols = len(cols) - len(staticcols)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
150
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
151 if not visible is None:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
152 for col in cols:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
153 if not col in visible:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
154 hiddencols.append(col)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
155
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
156 linkparse = [False] * len(cols)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
157
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
158 data = TupleDataset(name)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
159 for colidx in range(len(cols)):
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
160 col = cols[colidx]
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
161 autofilter = col in autofiltercols
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
162 hidden = col in hiddencols or colidx in hiddenindexes
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
163 data.columns.append(Column(col, autofilter=autofilter, hidden=hidden))
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
164
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
165 linkparse[colidx] = col in linkcols
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
166
3311
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
167 for row in self._read_rows(r):
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
168 row = map(lambda x: x.decode('utf-8'), row)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
169 if len(row) > num_entry_cols:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
170 row = row[:num_entry_cols]
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
171 elif len(row) < num_entry_cols:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
172 row.extend([''] * (num_entry_cols-len(row)))
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
173 row += staticvals
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
174 for colidx in range(len(row)):
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
175 item = row[colidx]
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
176 if linkparse[colidx]:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
177 try:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
178 url, item = item.split(' ', 1)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
179 if url == '':
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
180 display = escape(item)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
181 else:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
182 display = ''.join([
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
183 formatter.url(1, url=url),
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
184 formatter.text(item),
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
185 formatter.url(0)])
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
186 except ValueError:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
187 display = escape(item)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
188 else:
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
189 display = escape(item)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
190 row[colidx] = (display, item)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
191 data.addRow(tuple(row))
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
192 self.data = data
0
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
193
77665d8e2254 tag of nonpublic@localhost--archive/moin--enterprise--1.5--base-0
Thomas Waldmann <tw-public@gmx.de>
parents:
diff changeset
194 def format(self, formatter):
3311
fbcd580085de make new csv parser compatible with empty header line as in old
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 3309
diff changeset
195 browser = DataBrowserWidget(self.request, show_header=self._show_header)
3309
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
196 browser.setData(self.data)
4c14c613e275 new csv parser, uses data browser widget and supports much more, is backward compatible
Johannes Berg <johannes AT sipsolutions DOT net>
parents: 2286
diff changeset
197 self.request.write(browser.format())