view docs/CHANGES.fpletz @ 2064:5cc8f6714297

Add more event types. Return a list of messages generated by event handlers to caller.
author Karol 'grzywacz' Nowak <grzywacz@sul.uni.lodz.pl>
date Thu, 31 May 2007 01:00:27 +0200
parents 546ec510caa1
children
line wrap: on
line source
Branch moin/1.6-xapian-fpletz
=============================

  Known main issues:
    * Only term-based regex searching possible, modifier or heuristic to
      enable usage of _moinSearch for full compatibility?
    * HACK: MoinMoin.search.Xapian.Index._get_languages (wait for proper
      metadata)

  ToDo:
    * none

  ToDo (low priority):
    * Reevaluate Xapwrap, possibly drop it and rip out usable stuff
      (i.e. ExceptionTranslator)
    * Case-sensitive searches / Regexp on multiple terms: Graceful
      fallback to and/or merge with moinSearch based on nodes xapian can
      handle in the search term tree
      * currently, xapian will fetch relevant pages and feed those into
        _moinSearch for doing the real hard stuff it can't handle
      -> need for a query optimizer, after SoC?

  New Features:
    * Faster search thanks to Xapian
    * New searches:
        - LanguageSearch: language:de
        - CategorySearch: category:Homepage
        - MimetypeSearch: mimetype:image/png (for attachments/files)
        - DomainSearch: domain:underlay
        - History Search: available in advanced ui
      Note: Some currently only available when Xapian is used
    * New config options:
        xapian_search        0      enables xapian-powered search
        xapian_index_dir     None   directory for xapian indices
        xapian_stemming      1      toggles usage of stemmer, fallback
                                    to False if no stemmer installed
        search_results_per_page 10  determines how many hits should be
                                    shown on a fullsearch action
        xapian_index_history True   indexes all revisions of pages to
                                    allow searching in their history
  
  Bugfixes (only stuff that is buggy in moin/1.6 main branch):
    * ...

  Other Changes:
    * Some whitespace fixes in miscellaneous code
  
  Developer notes:
    * ...


Diary
=====

2006-06-10 Changed xapian_term() functions to return xapian.Query objects
but without touching the prefixes as we don't have a prefixMap yet. Will
implement this in MoinMoin.Xapian.Index. AndExpression needed some more
tweaking to use AND_NOT because Xapian doesn't provide a pure NOT. Should
be no issue with OrExpression as _moinSearch handles this correctly.

2006-06-11
    * Now handling prefixes correctly (title -> S, XLINKTO always with ':')

2006-06-15
    * Integrated basic stemming, english only for now (see issues).
    * Introduced LanguageSearch (new prefix lang/language)
    * Searching now works with stemmed terms but matching is limited due
      to usage of _moinSearch

2006-06-16
    * Indexing & searching now works without a stemmer installed (small
      bugfixes)

2006-06-17
    * Tackled some of the issues with matching stemmed words. Need some
      advice on how to detect and match them reliably using the current
      framework

2006-06-19
    * Introducing xapian_index_dir as a global directory for multiple
      xapian indices i.e. for wikifarms.

      Layout:
            xapian_index_dir/
                siteid1/
                    complete
                    index/
                    index-lock/
                    update-queue-lock/
                siteid2/
                    complete
                    index/
                    index-lock/
                    update-queue-lock/
                ...

        Possible extension: Xapian can handle multiple databases, maybe
        allow searching across defined wikis on a wikifarm
    * All stemming/matching issues resolved (hopefully)
    * Works now without xapian installed (enhance error reporting)

2006-06-21
    * Making stemming configurable (xapian_stemming) with fallback to
      False if no stemmer available
    * Xapian.use_stemming -> request.cfg.xapian_stemming
    * Fixed bug in the selection of the stemming language

2006-06-27
    * Splitting out MoinMoin/search.py to MoinMoin/search/*.py, no more
      need to invoke QueryParser manually when using searchPages, minor
      refactoring

2006-06-28
    * Abstraction of a locked search engine index:
      MoinMoin.search.builtin.BaseIndex, MoinMoin.search.Xapian.Index is
      derived from this, cleanups in calling structure and function
      prototypes to make it more extensible

2006-06-29
    * Tested some ideas with regexp searching

2006-07-01
    * Fully implemented term-based regexp searching

2006-07-04
    * Evaluating the current framework for the new UI (no new sane code to
      commit)

2006-07-05
    * Indexing correct positions in xapwrap

2006-07-06
    * Played with Xapian to get correct positions and where to integrate
      in MoinMoin

2006-07-07
    * Basic (quick and dirty, limitations and bugs included, but
      commit-ready) implementation of getting matches out of the Xapian DB

2006-07-08
    * No work: daytrip to Munich

2006-07-09
    * Bugfix for _moinSearch (not using Xapian)

2006-07-11
    * Make matches which we get from Xapian more reliable
    * Add TitleMatch support
    * Xapwrap needed some tuning (aka hacking), think about dropping
      and/or rewriting much of its code as it doesn't always fit (and
      probably won't in the future)

2006-07-12
2006-07-13
    * No work

2006-07-14
    * Minor bugfix for TitleMatch, now works correctly
    * First interesting match must be a TextMatch
    * Comment read_lock code from BaseIndex (should not be needed)
    * Support complete rebuild of the database (delete and add)

2006-07-17
    * SystemInfo macro now also shows if xapian is being used (index
      available) and more graceful fallback to moinSearch
    * Explored and evaluated the current framework for macros,
      formatters and stuff which we need to touch for the new search UI

2006-07-18
    * Fixed some bugs, whitespaces at EOL, better i18n for SystemInfo
    * Implemented paging support for searches, needs some style
      adjustments

2006-07-19
    * student didn't work on the project -- ThomasWaldmann

2006-07-20
    * Fixed some bugs found while testing regexp and case-sensitive searches
    * Conclusion after tinkering with the current code to allow
      cooperation between moinSearch and Xapian for case-sensitive
      searches (code buried): We probably need a rather big rewrite!

2006-07-21
2006-07-22
    * Final thoughts: No query optimizer for now. Case-sensitive
      sensitive search is done by querying Xapian with the lowercased
      terms and run _moinSearch over the relevant pages with the same
      query.
    * Indexing of categories

2006-07-23
    * CategorySearch is live
    * Subpage issue does not need changes: Can be done with regex magic
      I.e.: - subpages of MyPage: re:^MyPage/
            - subpages called SubPage: re:/SubPage
            - subpages called Subpage (1st level): re:[^/]*/SubPage
            - subpages called Subpage (last level): re:/Subpage$

2006-07-24
    * SystemInfo macro update (mtime)
    * nicer regexp support for TitleSearch

2006-07-25 .. 2006-08-01
    * student did not work on project

2006-08-02
    * Reformatted search statistics to use CSS and be more google-like
      (only in modern theme for now)
    * Added "search result info bar", showing revision, size, mtime,
      links for further searches (-> ToDo) etc.

2006-08-03 no work on project
2006-08-04 no work on project

2006-08-05 .. 2006-08-06
    * (finally :)) Google-like paging, using images from google.com until
      we get proper moin gfx
    * index domains of a page (standard, underlay)

2006-08-07
     * info bar for titlesearches
     * bugfix for results code: sometimes we never got a page instance
       in Found{Page,Attachment,...} which yielded strange errors

2006-08-08
    * added some more timers for regression testing
    * improved highlighting code to work better with stemming and
      special searches, extended SystemInfo macro

2006-08-09
    * use xapian for sorting, first step not to fetch all results
      -> still TODO: need real weight

2006-08-10
    * corrected range and count of results (estimated by xapian)
    * pagelinks only there are enough results

2006-08-10 .. 13 no work on project

2006-08-14
    * fixed some remaining issues with the ui

2006-08-15
    * removed Moooin gfx as requested by Google

2006-08-16 no work on project

2006-08-17
    * started advanced gui, new macro: AdvancedSearch

2006-08-18
    * eye-candy for advanced gui
    * reworked fullsearch action to work with AdvancedSearch and most of
      the 

2006-08-19
    * mimetype search works (more or less)
    * minor bugfixes (i18n etc.)
    * domain-specific search (underlay -> system pages)

2006-08-20
    * major fixes mimetype & mtime search and language sorting (neither
      really worked)

2006-08-21
    * indexing the history of pages (all revisions) if requested:
        xapian_index_history
    * implemented optional history search in advanced ui, defaults to off

SOC END

2006-08-22
    ...

2006-08-23 
    ...

2006-08-24 (on train from Hannover to Passau)
    * finish some code docs
    * added support for renaming, deleting pages (I somehow forgot :()
    * handle deleting of attachments correctly
    * testing of all this new stuff