XML. This requires installation of the library before building Swish-e. See XMLClassAttributes can specify a list of attribute names whose content is

Google

Swish-E Logo


CHANGES - List of revisions

Prev | Contents | Next

Table of Contents:

[ TOC ]

Revision History

This document contains list of bug fixes and feature additions to Swish-e.

[ TOC ]

Version 2.2.2 - November 14, 2002

Pass non- text/* files onto indexing code IF there is a FileFilter associated with the *extension* of the URL. Fixes the problem of not being able to index, say, pdf files by using the FileFilter configuation option.

Fixed bug where nulls were stripped when using FileFilter with -S prog. Catch by Greg Fenton. [moseley]

[ TOC ]

Version 2.2.1 - September 26, 2002

[ TOC ]

Version 2.2 - September 18, 2002

  • Default parser

    installed and DefaultContents or IndexContents is not used.

  • Selecting parsers

    built-in parsers.

  • SwishSpider and Filters

    Filters (FileFilter directive) did not work correctly when spidering with the -S http method. A new filter system was developed and now filtering of documents (e.g. pdf->html or MSWord->text) is handled by the src/SwishSpider program.

    When indexing with the -S http method only documents of content-type ``text/*'' are indexed. Other documents must be converted to text by using the filter system.

    indexed. [moseley]

  • configure script updates

    Updated from _WIN32 checks to feature checks using autoconf [moseley, norris]

  • updates to run on Alpha (Linux 2.4 (Debian 3.0))

    Fixed a cast error when calling zlib, and the calls to read/write a packed longs to disk. [jmruiz, moseley]

  • COALESCE_BUFFER_MAX_SIZE

    Some people were seeing the following error:

      
        err: Buffer too short in coalesce_word_locations.
    Increase COALESCE_BUFFER_MAX_SIZE in config.h and rebuild.

    This was due to indexing binary data or files with very large number of words. The best solution is to not index binary data or files with a very large number of words.

    Swish-e will now automatically reallocate the buffer as needed. [jmruiz]

[ TOC ]

Version 2.2rc1 - August 29, 2002

Many large changes were made internally in the code, some for performance reasons, some for feature changes and additions, and some to prepare for new features in later versions of Swish-e.

  • Documentation!

    Documentation is now included in the source distribution as .pod (perldoc) files, and as HTML files. In addition, the distribution can now generate PDF, postscript, and unix man pages from the source .pod files. See README for more information.

  • Indexing and searching speed

    The indexing process has been imporoved. Depending on a number of factors, you may see a significant improvement in indexing speed, especially if upgrading from version 1.x.

    Searching speed has also been improved. Properties are not loaded until results are displayed, and properties are pre-sorted during indexing to speed up sorting results by properties while searching.

  • Properties are written to a sepearte file

    Swish-e now stores document properties in a separate file. This means there are now two files that make up a Swish-e index. The default files are index.swish-e and index.swish-e.prop.

    This change frees memory while indexing, allowing larger collections to be indexed in memory.

  • Internal data stored as Properties

    Pre 2.2 some internal data was stored in fixed locations within the index, namely the file name, file size, and title. 2.2 introduced new internal data such as the last modified date, and document summaries. This data is considered meta data since it is data about a document.

    Instead of adding new data to the internal structure of the index file, it was decided to use the MetaNames and PropertyNames feature of Swish-e to store this meta information. This allows for new meta data to be added at a later time (e.g. Content-type), and provides an easy and customizable way to print results with the -p switch and the new -x switch. In addition, search results can now be sorted and limited by properties.

    For example, to sort by the rank and title:

      
        swish-e -w foo -s swishrank desc swishtitle asc

  • The header display has been slightly reorganized.

    If you are parsing output headers in a program then you may need to adjust your code. There's a new switch <-H> to control the level of header output when searching.

  • Results are now combined when searching more than one index.

    Swish-e now merges (and sorts) the results from multiple indexes when using -f to specify more than one index. This change effects the way maxhits (-m) works. Here's a summary of the way it works for the different versions.

      
        1.3.2 - MaxHits returns first N results starting from the first index.
            e.g. maxhits=20; 15 hits Index1, 40 hits Index2
            All 15 from Index1 plus first five from Index2 = 20 hits.

      
        2.0.0 - MaxHits returns first N results from each index.
            e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
            All 15 from Index1 plus 15 from Index2.

      
        2.2.0 - Results are merged and first N results are returned.
            e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
            Results are merged from each index and sorted
            (rank is the default sort) and only the first
            20 are returned.

  • New prog document source indexing method

    You can now use -S prog to use an external program to supply documents to Swish-e. This external program can be used to spider web servers, index it can be indexed by Swish-e. Examples are given in the prog-bin directory.

  • The indexing parser was rewritten to be more logical.

    TranslateCharacters now is done before WordCharacters is checked. For example,

      
        WordCharacters abcdefghijklmnopqrstuvwxyz
    TranslateCharacters ñ n

    Now El Niño will be indexed as El Nino (el and nino), even though ñ is not listed in WordCharacters.

    Previously, stopwords were checked after stemming and soundex conversions, as well as most of the other word checks (WordCharacters, min/max length and so on). This meant that the stopword list probably didn't work as expected when using stemming.

  • The search parser was rewritten to be more logical

    The search parser was rewritten to correct a number of logic errors. Swish-e did not differentiate between meta names, Swish-e operators and search words when parsing the query. This meant, for example, that metanames might be broken up by the WordCharacters setting, and that they could be stemmed.

    Swish-e operator characters "*()= can now be searched by escaping with a backslash. For example:

      
        ./swish-e -w 'this\=odd\)word'

    will end up searching for the word this=odd)word. To search for a backslash character preceed it with a backslash.

    Currently, searching for:

      
        ./swish-e -w 'this\*'

    is the same as a wildcard search. This may be fixed in the future.

    Searching for buzzwords with those characters will still require backslashing. This also may change to allow some un-escaped operator characters, but some will always need to be escaped (e.g. the double-quote phrase character).

  • Quotes and Backslash escapes in strings

    A bug was fixed in the parse_line() function (in string.c) where backslashes were not escaping the next character. parse_line() is used to parse a string of text into tokens (words). Normally splitting is done at whitespace. You may use quotes (single or double) to define a string (that might include whitespace) as a single parameter. The backslash can also be used to escape the following character when *within* quotes (e.g. to escape an embedded quote character).

      
        ReplaceRules append "foo bar"   <- define "foo bar" as a single word
    ReplaceRules append "foo\"bar"  <- escape the quotes
    ReplaceRules append 'foo"bar'   <- same thing

  • Example user.config file removed.

    Previous versions of Swish-e included a configuration file called user.config which contained examples of all directives. This has been replaced by a series of example configuration files located in the conf directory. The configuration directives are now described in SWISH-CONFIG.

  • Ports to Win32 and VMS

    David Norris has included the files required to build Swish-e under Windows. See src/win32. A self-extracting Windows version is available from the Download page of the swish-e.org web site.

    Jean-François Piéronne has provided the files required to build Swish-e under OpenVMS. See src/vms for more information.

  • String properties are concatenated

    Multiple string properties of the same name in a document are now concatenated into one property. A space character is added between the strings if needed. A warning will be generated if multiple numeric or date properties are found in the same document, and the additional properties will be ignored.

    Previously, properties of the same name were added to the index, but could not be retrieved.

    To do: remove the next pointer, and allow user-defined character to place between properties.

  • regex type added to ReplaceRules

    A more general purpose pattern replacement syntax.

  • New Parsers

    Swish-e's XML parser was replaced with James Clark's expat XML parser library.

    recommended for parsing HTML over Swish-e's internal HTML parser, and provides more features for both HTML and XML parsing.

  • Support for zlib

    Swish-e can be compiled with zlib. This is useful for compressing large properties. Building Swish-e with zlib is stronly recommended if you use its StoreDescription feature.

  • LST type of document no longer supported

    LST allowed indexing of files that contained multiple documents.

  • Temporary files

    To improve security Swish-e now uses the mkstemp(3) function to create temporary files. Temporary files are used while indexing only. This may result in some portability issues, but the security issues were overriding.

    (Currently this does not apply to the -S http indexing method.)

    mkstemp opens the temporary with O_EXCL|O_CREAT flags. This prevents overwriting existing files. In addition, the name of the file created is a lot harder to guess by attackers. The temporary file is created with only owner permissions.

    Please report any portability issues on the Swish-e discussion list.

  • Temporary file locations

    Swish-e now uses the environment variables TMPDIR, TMP, and TEMP (in that order) to decide where to write temporary files. The configuration setting of TmpDir will be used if none of the environment variables are set. Swish-e uses the current directory otherwise; there is no default temporary directory.

    Since the environment variables override the configuration settings, a warning will be issued if you set TmpDir in the configuration file and there's also an environment variable set.

    Temporary files begin with the letters ``swtmp'' (which can be changed in config.h), followed by two or more letters that indicate the type of temporary file, and some random characters to complete the file name. If indexing is aborted for some reason you may find these temporary files left behind.

  • New Fuzzy indexing method Double Metaphone

    Based on Lawrence Philips' Metaphone algorithm, add two new methods of creating a fuzzy index (in addition to Stemming and Soundex).

Changes to Configuration File Directives. Please see SWISH-CONFIG for more info.

  • New directives: IndexContents and DefaultContents

    The IndexContents directive assigns internal Swish-e document parsers to files based on their file type. The DefaultContents directive assigns a parser to be used on file that are not assigned a parser with IndexContents.

  • New directive: UndefinedMetaTags [error|ignore|index|auto]

    This describes what to do when a meta tag is found in a document that is not listed in the MetaNames directive.

  • New directive: IgnoreTags

    Will ignore text with the listed tags.

  • New directive: SwishProgParameters *list of words*

    Passes words listed to the external Swish-e program when running with -S prog document source method.

  • New directive: ConvertHTMLEntities [yes|no]

    Controls parsing and conversion of HTML entities.

  • New directive: DontBumpPositionOnMetaTags

    The word position is now bumped when a new metatag is found -- this is to prevent phrases from matching across meta tags. This directive will disable this behavior for the listed tags.

    This directive works for HTML and XML documents.

  • Changed directive: IndexComments

    This has been changed such that comments are not indexed by default.

  • Changed directive: IgnoreWords

    The builtin list of stopwords has been removed. Use of the SwishDefault word will generate a warning, and no stop words will be used. You must now specify a list of stopwords, or specify a file of stopwords.

    A sample file stopwords.txt has been included in the conf/stopwords directory of the distribution, and can be used by the directive:

      
        IgnoreWords File: /path/to/stopwords.txt

  • Change of the default for IgnoreTotalWordCountWhenRanking

    The default is now ``yes''.

  • New directive: Buzzwords

    Buzzwords are words that should be indexed as-is, without checking for stopwords, word length, WordCharacters, or any other of the word limiting features. This allows indexing of things like C++ when ``+'' is not listed in WordCharacters.

    Currenly, IgnoreFirstChar and IgnoreLastChar will be stripped before processing Buzzwords.

    In the future we may use separate IgnoreFirst/Last settings for buzzwords since, for example, you may wish to index all + within Swish-e words, but strip + from the start/end of Swish-e words, but not from the buzzword C++.

  • New directives: PropertyNamesNumeric PropertyNamesDate

    Before Swish-e 2.2 all user-defined document properties were stored in the index as strings. PropertyNamesNumeric and PropertyNamesDate tell it that a property should be stored in binary format. This allows for correct sorting of numeric properties.

    Currenly, only integers can be stored, such as a unix timestamp. (Swish-e uses strtoul to convert the number to an unsigned long internally.)

    PropertyNamesDate only indicates to Swish-e that a number is a unix timestamp, and to display the property as a formatted time when printing results. Swish does not currently parse date strings; you must provide a unix timestamp.

  • New directive: MetaNameAlias

    You may now create alias names for MetaNames. This allow you to map or group multiple names to the same MetaName.

  • New directive: PropertyNameAlias

    Creates aliases for a PropertyName.

  • New directive: PropertyNamesMaxLength

    Sets the max length of a text property.

  • New directive: HTMLLinksMetaName

    Defines a metaname to use for indexing href links in HTML documents.

  • New directive: ImageLinksMetaName

    Defines a metaname to use for indexing src links in <img> tags. Allow you to search image pathnames within HTML pages. Available only with

  • New directive: IndexAltTagMetaName

    parser.

  • New directive: AbsoluteLinks

    Attempts to convert relative links indexed with HTMLLinksMetaName and

  • New directive: ExtractPath

    Allows you to use a regular expression to extract out part of the path of each file and index it with a meta name. For example, this allows searches to be limited to parts of your file tree.

  • New directive: FileMatch

    FileMatch is similar to FileRules. Where FileRules is used to exclude files and directoires, FileMatch is used to include files.

  • New directive: PreSortedIndex

    Controls which properties are pre-sorted while indexing. All properties are sorted by default.

  • New directive: ParserWarnLevel

  • New directive: obeyRobotsNoIndex [yes|NO]

    NOINDEX.

      
        <meta name="robots" content="noindex">

    Also, comments may be used within HTML and XML source docs to block sections of content from indexing:

      
           <!-- SwishCommand noindex -->
       <!-- SwishCommand index -->

    and/or these may be used also:

      
           <!-- noindex -->
       <!-- index -->

  • New directive: UndefinedXMLAttributes

    This describes how the content of XML attributes should be indexed, if at all. This is similar to UndefinedMetaTags, but is only for XML attributes

  • New directive: XMLClassAttributes

    combined with the element name to form metanames.

  • New directive: PropCompressionLevel [0-9]

    If compiled with zlib, Swish-e uses this setting to control the level of compression applied to properties. Properties must be long enough (defined in config.h) to be compressed. Useful for StoreDescription.

  • Experimental directive: IgnoreNumberChars

    Defines a set of characters. If a word is made of of *only* those characters the word will not be indexed.

  • New directive: FuzzyIndexingMode

    This configuration directive is used to define the type of ``fuzzy'' index to create. Currently the options are:

      
        None
    Stemming
    Soundex
    Metaphone
    DoubleMetaphone

Changes to command line arguments. See SWISH-RUN for documentation on these switches.

[ TOC ]

Prev | Contents | Next


SWISH-E is distributed with no warranty under the terms of the GNU Public License,
Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA
Public questions may be posted to the SWISH-E Discussion.