annotate MoinMoin/scripts/packages/create_pagepacks.py @ 349:b0527dd2174c

Let the create_pagepacks read the page list from the local wiki and not from moinmaster. imported from: moin--main--1.5--patch-353
author Alexander Schremmer <alex@alexanderweb.de.tla>
date Fri, 30 Dec 2005 20:06:20 +0000
parents 14d0a26ab4ec
children be33ecd58d4e
rev   line source
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
1 #!/usr/bin/env python
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
2 # -*- coding: iso-8859-1 -*-
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
3 """
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
4 MoinMoin - Package Generator
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
5
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
6 @copyright: 2005 by Alexander Schremmer
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
7 @license: GNU GPL, see COPYING for details.
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
8 """
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
9
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
10 import os, sys
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
11 import zipfile
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
12 import threading
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
13 import xmlrpclib
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
14 from sets import Set
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
15 from datetime import datetime
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
16 from time import sleep
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
17
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
18 # your MoinMoin package path here
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
19 sys.path.insert(0, r"../../..")
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
20 sys.path.insert(0, r".")
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
21
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
22 from MoinMoin import config, wikidicts, wikiutil
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
23 from MoinMoin.Page import Page
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
24 from MoinMoin.PageEditor import PageEditor
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
25 from MoinMoin.request import RequestCLI
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
26 from MoinMoin.packages import packLine, unpackLine, MOIN_PACKAGE_FILE
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
27
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
28 master_url ="http://moinmaster.wikiwikiweb.de/?action=xmlrpc2"
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
29
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
30 EXTRA = u'extra'
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
31 NODIST = u'nodist'
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
32 ALL = u'all_languages'
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
33 COMPRESSION_LEVEL = zipfile.ZIP_STORED
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
34
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
35 def buildPageSets():
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
36 """ Calculates which pages should go into which package. """
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
37 pageSets = {}
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
38
349
b0527dd2174c Let the create_pagepacks read the page list from the local wiki and not from moinmaster.
Alexander Schremmer <alex@alexanderweb.de.tla>
parents: 316
diff changeset
39 #allPages = Set(xmlrpclib.ServerProxy(master_url).getAllPages())
b0527dd2174c Let the create_pagepacks read the page list from the local wiki and not from moinmaster.
Alexander Schremmer <alex@alexanderweb.de.tla>
parents: 316
diff changeset
40 allPages = Set(request.rootpage.getPageList())
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
41
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
42 systemPages = wikidicts.Group(request, "SystemPagesGroup").members()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
43
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
44 for pagename in systemPages:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
45 if pagename.endswith("Group"):
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
46 #print x + " -> " + repr(wikidicts.Group(request, x).members())
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
47 gd.addgroup(request, pagename)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
48
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
49 langPages = Set()
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
50 for name, group in gd.dictdict.items():
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
51 group.expandgroups(gd)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
52 groupPages = Set(group.members() + [name])
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
53 name = name.replace("SystemPagesIn", "").replace("Group", "")
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
54 pageSets[name] = groupPages
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
55 langPages |= groupPages
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
56
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
57 specialPages = Set(["SystemPagesGroup"])
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
58
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
59 masterNonSystemPages = allPages - langPages - specialPages
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
60
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
61 moinI18nPages = Set([x for x in masterNonSystemPages if x.startswith("MoinI18n")])
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
62
316
14d0a26ab4ec updated CHANGES, added MoinEditorPagesGroup to distributed pages
Thomas Waldmann <tw@waldmann-edv.de>
parents: 315
diff changeset
63 nodistPages = moinI18nPages | Set(["InterWikiMap", ])
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
64
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
65 extraPages = masterNonSystemPages - nodistPages
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
66
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
67 pageSets[ALL] = langPages
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
68
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
69 for name in pageSets.keys():
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
70 if name not in (u"English"):
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
71 pageSets[name] -= pageSets[u"English"]
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
72 pageSets[name] -= nodistPages
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
73
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
74 pageSets[EXTRA] = extraPages # stuff that maybe should be in some language group
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
75 pageSets[NODIST] = nodistPages # we dont want to have them in dist archive
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
76 return pageSets
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
77
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
78 def packagePages(pagelist, filename, function):
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
79 """ Puts pages from pagelist into filename and calls function on them on installation. """
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
80 try:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
81 os.remove(filename)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
82 except OSError:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
83 pass
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
84 zf = zipfile.ZipFile(filename, "w", COMPRESSION_LEVEL)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
85
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
86 cnt = 0
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
87 script = [packLine(['MoinMoinPackage', '1']),
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
88 ]
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
89
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
90 for pagename in pagelist:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
91 pagename = pagename.strip()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
92 page = Page(request, pagename)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
93 if page.exists():
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
94 cnt += 1
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
95 script.append(packLine([function, str(cnt), pagename]))
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
96 timestamp = wikiutil.version2timestamp(page.mtime_usecs())
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
97 zi = zipfile.ZipInfo(filename=str(cnt), date_time=datetime.fromtimestamp(timestamp).timetuple()[:6])
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
98 zi.compress_type = COMPRESSION_LEVEL
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
99 zf.writestr(zi, page.get_raw_body().encode("utf-8"))
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
100 else:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
101 #print >>sys.stderr, "Could not find the page %s." % pagename.encode("utf-8")
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
102 pass
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
103
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
104 script += [packLine(['Print', 'Installed MoinMaster page bundle %s.' % os.path.basename(filename)])]
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
105
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
106 zf.writestr(MOIN_PACKAGE_FILE, u"\n".join(script).encode("utf-8"))
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
107 zf.close()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
108
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
109 def removePages(pagelist):
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
110 """ Pages from pagelist get removed from the underlay directory. """
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
111 import shutil
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
112 for pagename in pagelist:
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
113 pagename = pagename.strip()
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
114 page = Page(request, pagename)
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
115 try:
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
116 underlay, path = page.getPageBasePath(-1)
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
117 shutil.rmtree(path)
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
118 except:
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
119 pass
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
120
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
121 def packageCompoundInstaller(bundledict, filename):
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
122 """ Creates a package which installs all other packages. """
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
123 try:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
124 os.remove(filename)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
125 except OSError:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
126 pass
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
127 zf = zipfile.ZipFile(filename, "w", COMPRESSION_LEVEL)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
128
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
129 script = [packLine(['MoinMoinPackage', '1']),
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
130 ]
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
131
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
132 script += [packLine(["InstallPackage", "SystemPagesSetup", name + ".zip"])
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
133 for name in bundledict.keys() if name not in (NODIST, EXTRA, ALL, u"English")]
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
134 script += [packLine(['Print', 'Installed all MoinMaster page bundles.'])]
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
135
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
136 zf.writestr(MOIN_PACKAGE_FILE, u"\n".join(script).encode("utf-8"))
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
137 zf.close()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
138
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
139 def getMasterPages():
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
140 """ Leechezzz. """
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
141 master = xmlrpclib.ServerProxy(master_url)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
142 maxThreads = 100
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
143
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
144 def downloadpage(wiki, pagename):
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
145 source = wiki.getPage(pagename)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
146 if source.find("##master-page:FrontPage") != -1:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
147 source += u"""\n\n||<tablestyle="background: lightyellow; width:100%; text-align:center">[[en]] If you want to add help pages in your favorite language, see '''SystemPagesSetup'''.||\n"""
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
148
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
149 PageEditor(request, pagename, uid_override="Fetching ...")._write_file(source)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
150 #print "Fetched " + pagename.encode("utf-8")
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
151
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
152 stopped = []
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
153 running = []
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
154
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
155 print "Loading master page list ..."
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
156 pagelist = master.getAllPages()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
157 print "Preparing threads ..."
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
158 for pagename in pagelist:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
159 t = threading.Thread(target=downloadpage, args=(master, pagename), name=pagename.encode("unicode_escape"))
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
160 stopped.append(t)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
161
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
162 print "Starting scheduler ..."
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
163 while len(running) > 0 or len(stopped) != 0:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
164 for x in running:
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
165 if not x.isAlive():
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
166 #print "Found dead thread " + repr(x)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
167 running.remove(x)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
168 print "running %i| stopped %i" % (len(running), len(stopped))
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
169 for i in xrange(min(maxThreads - len(running), len(stopped))):
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
170 t = stopped.pop()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
171 running.append(t)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
172 t.start()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
173 #print "Scheduled %s." % repr(t)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
174 sleep(1)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
175
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
176 request = RequestCLI(url='localhost/')
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
177 request.form = request.args = request.setup_args()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
178
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
179 gd = wikidicts.GroupDict(request)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
180 gd.reset()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
181
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
182 #getMasterPages()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
183 print "Building page sets ..."
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
184 pageSets = buildPageSets()
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
185
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
186 print "Creating packages ..."
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
187 generate_filename = lambda name: os.path.join('testwiki', 'underlay', 'pages', 'SystemPagesSetup', 'attachments', '%s.zip' % name)
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
188
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
189 packageCompoundInstaller(pageSets, generate_filename(ALL))
315
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
190
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
191 [packagePages(list(pages), generate_filename(name), "ReplaceUnderlay")
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
192 for name, pages in pageSets.items() if not name in (u'English', ALL, NODIST)]
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
193
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
194 [removePages(list(pages))
830cf8bdcf24 implemented 'make pagepacks', updated underlay
Thomas Waldmann <tw@waldmann-edv.de>
parents: 313
diff changeset
195 for name, pages in pageSets.items() if not name in (u'English', ALL)]
313
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
196
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
197 print "Finished."
e341b54060c3 some minor changes to create_pagepacks
Thomas Waldmann <tw@waldmann-edv.de>
parents:
diff changeset
198