view docs/CHANGES.fpletz @ 921:45e286183872

abstraction work on search engine index & cleanups
author Franz Pletz <fpletz AT franz-pletz DOT org>
date Wed, 28 Jun 2006 23:41:46 +0200
parents 5469c8b911a4
children 22f6f589162a
line wrap: on
line source
Branch moin/1.6-xapian-fpletz
=============================

  Known main issues:
    * Regex searching with Xapian?
    * HACK: MoinMoin.Xapian.Index._get_languages (wait for proper metadata)

  ToDo:
    * Mockup the new search UI
    * Write/update documentation for all the new search stuff
    * Indexing and searching of categories (new term prefix)
    * Drop _moinSearch when using Xapian and use term positions provided
      by Xapian itself, needs some reworking of WikiAnalyzer/xapwrap to
      get the position of stemmed words right

  New Features:
    * Faster search thanks to Xapian
    * Searching for languages with new prefix lang/language, i.e. lang:de
      Note: Currently only available when Xapian is used
    * New config options:
        xapian_search        0      enables xapian-powered search
        xapian_index_dir     None   directory for xapian indices
        xapian_stemming      True   Toggles usage of stemmer, fallback
                                    to False if no stemmer installed
  
  Bugfixes (only stuff that is buggy in moin/1.6 main branch):
    * ...

  Other Changes:
    * ...
  
  Developer notes:
    * ...


Diary
=====

2006-06-10 Changed xapian_term() functions to return xapian.Query objects
but without touching the prefixes as we don't have a prefixMap yet. Will
implement this in MoinMoin.Xapian.Index. AndExpression needed some more
tweaking to use AND_NOT because Xapian doesn't provide a pure NOT. Should
be no issue with OrExpression as _moinSearch handles this correctly.

2006-06-11
    * Now handling prefixes correctly (title -> S, XLINKTO always with ':')

2006-06-15
    * Integrated basic stemming, english only for now (see issues).
    * Introduced LanguageSearch (new prefix lang/language)
    * Searching now works with stemmed terms but matching is limited due
      to usage of _moinSearch

2006-06-16
    * Indexing & searching now works without a stemmer installed (small
      bugfixes)

2006-06-17
    * Tackled some of the issues with matching stemmed words. Need some
      advice on how to detect and match them reliably using the current
      framework

2006-06-19
    * Introducing xapian_index_dir as a global directory for multiple
      xapian indices i.e. for wikifarms.

      Layout:
            xapian_index_dir/
                siteid1/
                    complete
                    index/
                    index-lock/
                    update-queue-lock/
                siteid2/
                    complete
                    index/
                    index-lock/
                    update-queue-lock/
                ...

        Possible extension: Xapian can handle multiple databases, maybe
        allow searching across defined wikis on a wikifarm
    * All stemming/matching issues resolved (hopefully)
    * Works now without xapian installed (enhance error reporting)

2006-06-21
    * Making stemming configurable (xapian_stemming) with fallback to
      False if no stemmer available
    * Xapian.use_stemming -> request.cfg.xapian_stemming
    * Fixed bug in the selection of the stemming language

2006-06-27
    * Splitting out MoinMoin/search.py to MoinMoin/search/*.py, no more
      need to invoke QueryParser manually when using searchPages, minor
      refactoring

2006-06-28
    * Abstraction of a locked search engine index:
      MoinMoin.search.builtin.BaseIndex, MoinMoin.search.Xapian.Index is
      derived from this, cleanups in calling structure and function
      prototypes to make it more extensible