changeset 2180:80fc914af5c8

integrated CHANGES.fpletz into main CHANGES, TODOs/unclear stuff went to MoinMoin:XapianIntegration
author Thomas Waldmann <tw AT waldmann-edv DOT de>
date Wed, 20 Jun 2007 18:02:10 +0200
parents e81af778f99d
children 340ae0e7529e
files docs/CHANGES docs/CHANGES.fpletz
diffstat 2 files changed, 31 insertions(+), 313 deletions(-) [+]
line wrap: on
line diff
--- a/docs/CHANGES	Thu Apr 19 09:42:17 2007 +0200
+++ b/docs/CHANGES	Wed Jun 20 18:02:10 2007 +0200
@@ -49,36 +49,6 @@
       many themes!
 
 
-Branch moin-1.6-xapian:
-  /!\ This needs to be merged together with Franz' CHANGES file into the section
-      below.
-
-  New Features:
-    * Added Xapian (see http://xapian.org/) based indexed search code.
-      Our implementation is still buggy, only use it if you want to help
-      debugging it or to implement / test indexing filters (see
-      MoinMoin/filter/). To use this:
-      * Install xapian-core and xapian-bindings on your machine.
-        We used 0.9.4, but newer code should hopefully work, too.
-      * cfg.xapian_search = True
-      * Execute this to build the index:
-        $ moin ... index build   # indexes pages and attachments
-        $ moin ... index build --files=files.lst  # same plus a list of files
-        You should run those commands as the same user you use for your wiki,
-        usually this is the webserver userid, e.g.:
-        $ sudo -u www-data moin --config=... --wiki-url=wiki.example.org/ \
-               index build --files=files.lst
-  ToDo:
-    * fix/improve query parsing (xapian_term member functions)
-    * fix/improve evaluation of search result
-    * maybe add some "xapian native query" mode (can we make it working without
-      the _moinSearch post-processing (not possible as it uses the same query))
-
-  Other Changes:
-    * Removed Lupy based indexed search code. If you were brave enough to
-      use cfg.lupy_search, you maybe want to try cfg.xapian_search instead.
-
-
 Version 1.6.current:
 
   Developer notes (these should be moved to the end in the release):
@@ -182,6 +152,35 @@
       'quicklinks' which will be lost when downgrading.)
 
   New Features:
+    * Added Xapian (see http://xapian.org/) based indexed search code.
+      To use this:
+      * Install xapian-core and xapian-bindings on your machine.
+        We used 0.9.4, but newer code should hopefully work, too.
+      * cfg.xapian_search = True
+      * Execute this to build the index:
+        $ moin ... index build   # indexes pages and attachments
+        $ moin ... index build --files=files.lst  # same plus a list of files
+        You should run those commands as the same user you use for your wiki,
+        usually this is the webserver userid, e.g.:
+        $ sudo -u www-data moin --config=... --wiki-url=wiki.example.org/ \
+               index build --files=files.lst
+    * New searches:
+        - LanguageSearch: language:de
+        - CategorySearch: category:Homepage
+        - MimetypeSearch: mimetype:image/png (for attachments/files)
+        - DomainSearch: domain:underlay or domain:standard
+        - History Search: available in advanced ui
+      Note: Some currently only available when Xapian is used
+    * New config options:
+        xapian_search        0      enables xapian-powered search
+        xapian_index_dir     None   directory for xapian indices
+                                    (can be shared for wiki farms)
+        xapian_stemming      1      toggles usage of stemmer, fallback
+                                    to False if no stemmer installed
+        search_results_per_page 10  determines how many hits should be
+                                    shown on a fullsearch action
+        xapian_index_history True   indexes all revisions of pages to
+                                    allow searching in their history
     * The action menu now calls the actions for the revision of the page you
       are currently viewing. If you are viewing the current page revision, you
       get the same behaviour as in moin 1.5, but if you are viewing an old
@@ -509,6 +508,8 @@
       Reasons: it had trouble with FCKEditor and was not updated since a longer
                time. OTOH, Microsoft released IE7. And there are lots of
                other and better browsers anyway.
+    * Removed Lupy based indexed search code. If you were brave enough to
+      use cfg.lupy_search, you maybe want to try cfg.xapian_search instead.
 
 
 Version 1.5.current:
--- a/docs/CHANGES.fpletz	Thu Apr 19 09:42:17 2007 +0200
+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
@@ -1,283 +0,0 @@
-Branch moin/1.6-xapian-fpletz
-=============================
-
-  Known main issues:
-    * Only term-based regex searching possible, modifier or heuristic to
-      enable usage of _moinSearch for full compatibility?
-    * HACK: MoinMoin.search.Xapian.Index._get_languages (wait for proper
-      metadata)
-
-  ToDo:
-    * none
-
-  ToDo (low priority):
-    * Reevaluate Xapwrap, possibly drop it and rip out usable stuff
-      (i.e. ExceptionTranslator)
-    * Case-sensitive searches / Regexp on multiple terms: Graceful
-      fallback to and/or merge with moinSearch based on nodes xapian can
-      handle in the search term tree
-      * currently, xapian will fetch relevant pages and feed those into
-        _moinSearch for doing the real hard stuff it can't handle
-      -> need for a query optimizer, after SoC?
-
-  New Features:
-    * Faster search thanks to Xapian
-    * New searches:
-        - LanguageSearch: language:de
-        - CategorySearch: category:Homepage
-        - MimetypeSearch: mimetype:image/png (for attachments/files)
-        - DomainSearch: domain:underlay
-        - History Search: available in advanced ui
-      Note: Some currently only available when Xapian is used
-    * New config options:
-        xapian_search        0      enables xapian-powered search
-        xapian_index_dir     None   directory for xapian indices
-        xapian_stemming      1      toggles usage of stemmer, fallback
-                                    to False if no stemmer installed
-        search_results_per_page 10  determines how many hits should be
-                                    shown on a fullsearch action
-        xapian_index_history True   indexes all revisions of pages to
-                                    allow searching in their history
-  
-  Bugfixes (only stuff that is buggy in moin/1.6 main branch):
-    * ...
-
-  Other Changes:
-    * Some whitespace fixes in miscellaneous code
-  
-  Developer notes:
-    * ...
-
-
-Diary
-=====
-
-2006-06-10 Changed xapian_term() functions to return xapian.Query objects
-but without touching the prefixes as we don't have a prefixMap yet. Will
-implement this in MoinMoin.Xapian.Index. AndExpression needed some more
-tweaking to use AND_NOT because Xapian doesn't provide a pure NOT. Should
-be no issue with OrExpression as _moinSearch handles this correctly.
-
-2006-06-11
-    * Now handling prefixes correctly (title -> S, XLINKTO always with ':')
-
-2006-06-15
-    * Integrated basic stemming, english only for now (see issues).
-    * Introduced LanguageSearch (new prefix lang/language)
-    * Searching now works with stemmed terms but matching is limited due
-      to usage of _moinSearch
-
-2006-06-16
-    * Indexing & searching now works without a stemmer installed (small
-      bugfixes)
-
-2006-06-17
-    * Tackled some of the issues with matching stemmed words. Need some
-      advice on how to detect and match them reliably using the current
-      framework
-
-2006-06-19
-    * Introducing xapian_index_dir as a global directory for multiple
-      xapian indices i.e. for wikifarms.
-
-      Layout:
-            xapian_index_dir/
-                siteid1/
-                    complete
-                    index/
-                    index-lock/
-                    update-queue-lock/
-                siteid2/
-                    complete
-                    index/
-                    index-lock/
-                    update-queue-lock/
-                ...
-
-        Possible extension: Xapian can handle multiple databases, maybe
-        allow searching across defined wikis on a wikifarm
-    * All stemming/matching issues resolved (hopefully)
-    * Works now without xapian installed (enhance error reporting)
-
-2006-06-21
-    * Making stemming configurable (xapian_stemming) with fallback to
-      False if no stemmer available
-    * Xapian.use_stemming -> request.cfg.xapian_stemming
-    * Fixed bug in the selection of the stemming language
-
-2006-06-27
-    * Splitting out MoinMoin/search.py to MoinMoin/search/*.py, no more
-      need to invoke QueryParser manually when using searchPages, minor
-      refactoring
-
-2006-06-28
-    * Abstraction of a locked search engine index:
-      MoinMoin.search.builtin.BaseIndex, MoinMoin.search.Xapian.Index is
-      derived from this, cleanups in calling structure and function
-      prototypes to make it more extensible
-
-2006-06-29
-    * Tested some ideas with regexp searching
-
-2006-07-01
-    * Fully implemented term-based regexp searching
-
-2006-07-04
-    * Evaluating the current framework for the new UI (no new sane code to
-      commit)
-
-2006-07-05
-    * Indexing correct positions in xapwrap
-
-2006-07-06
-    * Played with Xapian to get correct positions and where to integrate
-      in MoinMoin
-
-2006-07-07
-    * Basic (quick and dirty, limitations and bugs included, but
-      commit-ready) implementation of getting matches out of the Xapian DB
-
-2006-07-08
-    * No work: daytrip to Munich
-
-2006-07-09
-    * Bugfix for _moinSearch (not using Xapian)
-
-2006-07-11
-    * Make matches which we get from Xapian more reliable
-    * Add TitleMatch support
-    * Xapwrap needed some tuning (aka hacking), think about dropping
-      and/or rewriting much of its code as it doesn't always fit (and
-      probably won't in the future)
-
-2006-07-12
-2006-07-13
-    * No work
-
-2006-07-14
-    * Minor bugfix for TitleMatch, now works correctly
-    * First interesting match must be a TextMatch
-    * Comment read_lock code from BaseIndex (should not be needed)
-    * Support complete rebuild of the database (delete and add)
-
-2006-07-17
-    * SystemInfo macro now also shows if xapian is being used (index
-      available) and more graceful fallback to moinSearch
-    * Explored and evaluated the current framework for macros,
-      formatters and stuff which we need to touch for the new search UI
-
-2006-07-18
-    * Fixed some bugs, whitespaces at EOL, better i18n for SystemInfo
-    * Implemented paging support for searches, needs some style
-      adjustments
-
-2006-07-19
-    * student didn't work on the project -- ThomasWaldmann
-
-2006-07-20
-    * Fixed some bugs found while testing regexp and case-sensitive searches
-    * Conclusion after tinkering with the current code to allow
-      cooperation between moinSearch and Xapian for case-sensitive
-      searches (code buried): We probably need a rather big rewrite!
-
-2006-07-21
-2006-07-22
-    * Final thoughts: No query optimizer for now. Case-sensitive
-      sensitive search is done by querying Xapian with the lowercased
-      terms and run _moinSearch over the relevant pages with the same
-      query.
-    * Indexing of categories
-
-2006-07-23
-    * CategorySearch is live
-    * Subpage issue does not need changes: Can be done with regex magic
-      I.e.: - subpages of MyPage: re:^MyPage/
-            - subpages called SubPage: re:/SubPage
-            - subpages called Subpage (1st level): re:[^/]*/SubPage
-            - subpages called Subpage (last level): re:/Subpage$
-
-2006-07-24
-    * SystemInfo macro update (mtime)
-    * nicer regexp support for TitleSearch
-
-2006-07-25 .. 2006-08-01
-    * student did not work on project
-
-2006-08-02
-    * Reformatted search statistics to use CSS and be more google-like
-      (only in modern theme for now)
-    * Added "search result info bar", showing revision, size, mtime,
-      links for further searches (-> ToDo) etc.
-
-2006-08-03 no work on project
-2006-08-04 no work on project
-
-2006-08-05 .. 2006-08-06
-    * (finally :)) Google-like paging, using images from google.com until
-      we get proper moin gfx
-    * index domains of a page (standard, underlay)
-
-2006-08-07
-     * info bar for titlesearches
-     * bugfix for results code: sometimes we never got a page instance
-       in Found{Page,Attachment,...} which yielded strange errors
-
-2006-08-08
-    * added some more timers for regression testing
-    * improved highlighting code to work better with stemming and
-      special searches, extended SystemInfo macro
-
-2006-08-09
-    * use xapian for sorting, first step not to fetch all results
-      -> still TODO: need real weight
-
-2006-08-10
-    * corrected range and count of results (estimated by xapian)
-    * pagelinks only there are enough results
-
-2006-08-10 .. 13 no work on project
-
-2006-08-14
-    * fixed some remaining issues with the ui
-
-2006-08-15
-    * removed Moooin gfx as requested by Google
-
-2006-08-16 no work on project
-
-2006-08-17
-    * started advanced gui, new macro: AdvancedSearch
-
-2006-08-18
-    * eye-candy for advanced gui
-    * reworked fullsearch action to work with AdvancedSearch and most of
-      the 
-
-2006-08-19
-    * mimetype search works (more or less)
-    * minor bugfixes (i18n etc.)
-    * domain-specific search (underlay -> system pages)
-
-2006-08-20
-    * major fixes mimetype & mtime search and language sorting (neither
-      really worked)
-
-2006-08-21
-    * indexing the history of pages (all revisions) if requested:
-        xapian_index_history
-    * implemented optional history search in advanced ui, defaults to off
-
-SOC END
-
-2006-08-22
-    ...
-
-2006-08-23 
-    ...
-
-2006-08-24 (on train from Hannover to Passau)
-    * finish some code docs
-    * added support for renaming, deleting pages (I somehow forgot :()
-    * handle deleting of attachments correctly
-    * testing of all this new stuff
-