CHANGES -- List of revisions
This document contains list of bug fixes and feature additions to Swish-e.
http://bugzilla.gnome.org/show_bug.cgi?id=153937
IgnoreWords \
the \
am \
is \
are \
was
UseStemming was coded as an alias for FuzzyIndexingMode when Snowball was compiled in (the default), but "no" doesn't always mean no when the Norwegian stemmer is available.
Fixed compile problem with building incremental indexing mode. This is an experimental option with swish-e to allow adding files to an index. See configure --help for build option. Incremental indexes are not compatible with standard indexes.
Added a few comments about use of CPPFLAGS and LDFLAGS.
Updated to work with latest version of hypermail (pre-2.1.9).
Format string for generating date did not include the time zone in location. Add strftime format string to config.h
Fixed a few problems with printing properties:
1) Using -p and -x showed different results if a bad property value was given:
$ swish-e -w not dkdk -p badname -H0
err: Unknown Display property name "badname"
.
$ swish-e -w not dkdk -x '<badname>\n' -H0
(NULL)
Now both return an error.
2) Fixed bug where using a "fmt" string with -x output generated (bad) output if the result did not have the specified property.
$ swish-e -w not dkdk -x '<somedate>\n' -H0 # undefined value
$ swish-e -w not dkdk -x '<somedate fmt="%Y %B %d">\n' -H0
%Y %B 1075353525
Now nothing is printed if the property does not exist.
3) Updated SWISH::API to croak() on invalid property names, and to return undefined values for missing properties.
4) Updated swish.cgi and search.cgi to not generate warnings on undefined values return as properties. Note that swish.cgi will now die on undefined properties. Previously would just display (NULL).
Parser.c was incorrectly calling warning() incorrectly. And -Wall was not catching this!
Parser was not checking for use of Swish-e reserved property names.
<swishrank>foo</swishrank>
This will now generate a warning.
search.cgi is a new skeleton CGI script that uses SWISH::API for searching. It is installed in the same location as swish.cgi.
Added a number of functions to the C API (and SWISH::API) to access the stemmer used when indexing a given index.
Added commas to summary display at end of indexing.
Parser.c was updated to flush the text buffer before and after every (non-inline HTML) tag.
The problem was that:
foo<tag>bar</tag>baz
would index as a single word "foobarbaz".
DirTree.pl was updated to work with SWISH::Filter and to work on Windows. DirTree.pl is a program to fetch files from the file system and works with the -S prog input method.
Fixed configure script to build incremental option. Note that this is still experimental. But testers are welcome.
Mark Fletcher with the help of valgrind found a bug in headers.c function SwishIndexHeaderNames used by the C API.
At the prompting of Doralyn Rossmann updated SEARCH.pod to try and make the explanation of searching clearer, and to fix an error in the description of nested searches.
Swish-e version 2.4.0 has a different index file format from previous versions of Swish-e. Upgrading will require reindexing -- version 2.4.0 cannot read indexes created with previous versions.
When using -S http method robots.txt was not closed and that caused the (last) .contents file to not be unlinked under Windows. Windows seems to think filenames are related to files.
SWISH::Filter now scans \$libexecdir in addition to the PATH for programs (such at catdoc and pdftotext), and also checks for programs by adding the extensions ".exe" and ".bat" to the program name.
The sample templates included with swish.cgi are now installed in \$pkgdatadir (typically /usr/local/share/swish-e).
Fixed bug in query parser caused in rc2's (pr2) attempt to catch wildcards errors.
Fixed a problem when these were used in combination:
MetaNames swishtitle
MetaNameAlias swishtitle title
That failed to correctly reset the metaname stack and indexed text under the wrong metaID.
Due to the way the query parser "works" a search of
"foo *"
would result in a search of "foo*". Now that results in:
err: Single wildcard not allowed as word
Brad Miele reported that the word "andes" was not being found. It was being stemmed to "and" when was then considered an operator. [moseley]
PropertyNamesSortKeyLength sets the sort key length to use when sorting string properties. The default is 100 characters. There was a hard-coded 100 char limit before, but that was a problem where people were not building from source (Windows). The value of this is questionable -- it's intended to limit how much memory is used when sorting while indexing and searching. [moseley]
Reworked much of the sorting code. Still to do is setting the character sort order. [moseley]
Fixed leak of not releasing memory of index file name and swish_handle destroy, and fixed SwishStemWord to default to the Stemmer_en. [moseley]
Fixed libtest.c example program that was not cleaning up memory after an error condition.
Swish-e now has support for Snowball stemmers (http://snowball.tartarus.org/). The stemmers are enabled for an index with FuzzyIndexingMode Stemming_* where "*" can be:
de, dk, en1, en2, es, fi, fr, it, nl, no, pt, ru, se
In addition, UseStemming yes or FuzzyIndexingMode Stemming_en will use the old stemmer.
The swish.cgi script was not correctly escaping HTML when searching by the right combination of metanames and highlighting module. This could lead to cross-site scripting if indexing un-trusted documents. [moseley]
To build as a .deb unpack the distribution and chdir then run
$ fakeroot debian/build binary
Then install the generated .deb file with dpkg -i
spider.pl is installed in the libexecdir directory as well as the SWISH::Filter modules. PDF, MS Word, MP3, and XML documents will be indexed automatically if the required helper applications (e.g. catdoc, pdftotext) or scripts (e.g. MP3::Tag) are installed.
Swish also knows about libexecdir, so you you specify a relative path with -S prog swish-e will look for the program in libexecdir. This is mostly for spider.pl so indexing only requires:
IndexDir spider.pl
SwishProgParameters default http://localhost/index.html
And swish-e will find spider.pl and SWISH::Filter will be used to convert docs.
Document-Type was not being reset after set input from a -S prog program causing the wrong parser to be used. [moseley]
Swish replaces all series of low ASCII chars with a single space character. This option instructs swish to store all chars in the property. [moseley]
Defaults used with -S http access method were changed.
Delay was reduced from one minute between start of each request to five seconds between requests.
MaxDepth was changed from five to zero, meaning there is no limit to depth indexed by default. [moseley]
The swishspider program is now installed in \$prefix/lib/swish-e by default. This can be changed by the --libexecdir option to configure.
The SpiderDirectory option now defaults to the value of libexecdir instead of the current directory. [moseley]
Replaces the build system with Autotools. Now builds libswish-e as a shared library on systems that support shared libraries. The swish-e binary links against this shared library. Can also build outside the source tree on platforms with GNU make. [moseley]
Running "make install" now installs additional files. Files include the swish-e binary, the libswish-e search library, swish-e.h header, documentation files, the swishspider program, and Perl modules used for the example swish.cgi search script. Directories will be created if they do not already exist. Installation directories can be specified at build time.
Swish was not correctly detecting the end of the inverted index when searching a wildcard word that was past the last word in the index. Caught by Frank Heasley. [moseley]
The setting MAX_SORT_STRING_LEN in src/config.h sets the max length used when sorting in swish-e. You may reduce this number to save memory while sorting, or increase it if you have very long properties to sort.
The -p option to print properties was escaping double quotes in properties with the &quot; entity. -x does not do that, so inconsistent. -p no longer converts double quotes. The user should pick a good delimiter with -d or preferably use the -x method for generating output.
The XML parser was being passed the incorrect buffer length when used on Windows platform causing the parser to abort with an error.
SWISH-E versions starting with 2.3.4 use kernel version numbering. Versions are in the form: Major.Minor.Build. Odd minor versions are development. Even minor versions are releases. 2.3.4 would be a development version. 2.4.0 would be a release version. 2.3.20 would be the 20th build of 2.3.
RPMs can be built with:
./configure
make dist
Copy the resulting tarball to RPM's SOURCES directory and then run as a superuser:
rpmbuild -ba rpm/swish-e.spec
You should have swish-e packages in your RPMS/<arch> directory. [augur]
Most perl scripts provided with SWISH-E now use /usr/bin/perl by default. Note that some scripts are generated at build time, so those will look in the path for the location of the perl binary.
MetaNamesRank can be used to adjust the ranking for words based on the word's MetaName.
The Swish-e C library interface was rewritten to provide better memory management and better separation of data. Most indexing related code has been removed from the library. A new header file is provided for the API: swish-e.h.
The Perl module SWISHE was replaced with the SWISH::API module in the Swish-e distribution.
Previous versions of the SWISHE module will not work with this version of Swish-e.
If you are using the SWISHE module from a previous version of Swish then you must either rewrite your code to use the new SWISH::API module (highly recommended) or use the replacement SWISHE module. The replacement SWISHE module is a thin interface to the SWISH::API module. It can be downloaded from
http://swish-e.org/Download/old/SWISHE-0.03.tar.gz
Corrected problem when using NoContents with binary files and the HTML2 parser.
Trying to index image file names with:
IndexOnly .gif .jpeg
NoContents .gif .jpeg
failed to index the path names because the default parser (HTML2 when libxml2 is linked with swish-e) was not finding any text in the binary files. [moseley]
The example/swish.cgi script can now use the SWISH::API module for searching an index. Combined with mod_perl this module can improve search performance considerably.
The Perl modules used with the swish.cgi script have all been moved into the SWISH::* namespace. Hence, files in the modules directory were moved into the modules::SWISH directory.
Multiple -L options were ORing instead of ANDing. Catch by Patrick Mouret. [moseley]
Pass non- text/* files onto indexing code IF there is a FileFilter associated with the *extension* of the URL. Fixes the problem of not being able to index, say, pdf files by using the FileFilter configuation option.
Fixed bug where nulls were stripped when using FileFilter with -S prog. Catch by Greg Fenton. [moseley]
Failed to use the correct default parser when using the No-Contents header and libxml2 linked in. [moseley]
8-byte alignment in mem_zones is is required for these machine [moseley]
Was not correctly removing words from index when parser aborted [jmruiz]
Fixed segfault caused by trying to print null dates while merging duplicate files. [moseley]
Spelling corrections to the SWISH-CONFIG pod page [Steve Eckert]
Fixed a zlib test error that used "==" in a test [Steve Eckert]
The VMS build was updated [Jean-François PIÉRONNE]
Added missing filters and vms build file into MANIFEST [moseley]
Swish-e will now use the HTML2 (libxml2) parser by default if libxml2 is installed and DefaultContents or IndexContents is not used.
Allow HTML*, XML*, and TXT* to automatically select the libxml2-based parsers if libxml2 is linked with Swish-e, otherwise fallback to the built-in parsers.
Filters (FileFilter directive) did not work correctly when spidering with the -S http method. A new filter system was developed and now filtering of documents (e.g. pdf->html or MSWord->text) is handled by the src/SwishSpider program.
When indexing with the -S http method only documents of content-type "text/*" are indexed. Other documents must be converted to text by using the filter system.
Fixed bug in xml.c reported by Rodney Barnett when very long words were indexed. [moseley]
Updated from _WIN32 checks to feature checks using autoconf [moseley, norris]
Fixed a cast error when calling zlib, and the calls to read/write a packed longs to disk. [jmruiz, moseley]
Some people were seeing the following error:
err: Buffer too short in coalesce_word_locations.
Increase COALESCE_BUFFER_MAX_SIZE in config.h and rebuild.
This was due to indexing binary data or files with very large number of words. The best solution is to not index binary data or files with a very large number of words.
Swish-e will now automatically reallocate the buffer as needed. [jmruiz]
Many large changes were made internally in the code, some for performance reasons, some for feature changes and additions, and some to prepare for new features in later versions of Swish-e.
Documentation is now included in the source distribution as .pod (perldoc) files, and as HTML files. In addition, the distribution can now generate PDF, postscript, and unix man pages from the source .pod files. See README for more information.
The indexing process has been imporoved. Depending on a number of factors, you may see a significant improvement in indexing speed, especially if upgrading from version 1.x.
Searching speed has also been improved. Properties are not loaded until results are displayed, and properties are pre-sorted during indexing to speed up sorting results by properties while searching.
Swish-e now stores document properties in a separate file. This means there are now two files that make up a Swish-e index. The default files are index.swish-e and index.swish-e.prop.
This change frees memory while indexing, allowing larger collections to be indexed in memory.
Pre 2.2 some internal data was stored in fixed locations within the index, namely the file name, file size, and title. 2.2 introduced new internal data such as the last modified date, and document summaries. This data is considered meta data since it is data about a document.
Instead of adding new data to the internal structure of the index file, it was decided to use the MetaNames and PropertyNames feature of Swish-e to store this meta information. This allows for new meta data to be added at a later time (e.g. Content-type), and provides an easy and customizable way to print results with the -p switch and the new -x switch. In addition, search results can now be sorted and limited by properties.
For example, to sort by the rank and title:
swish-e -w foo -s swishrank desc swishtitle asc
If you are parsing output headers in a program then you may need to adjust your code. There's a new switch <-H> to control the level of header output when searching.
Swish-e now merges (and sorts) the results from multiple indexes when using -f to specify more than one index. This change effects the way maxhits (-m) works. Here's a summary of the way it works for the different versions.
1.3.2 - MaxHits returns first N results starting from the first index.
e.g. maxhits=20; 15 hits Index1, 40 hits Index2
All 15 from Index1 plus first five from Index2 = 20 hits.
2.0.0 - MaxHits returns first N results from each index.
e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
All 15 from Index1 plus 15 from Index2.
2.2.0 - Results are merged and first N results are returned.
e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
Results are merged from each index and sorted
(rank is the default sort) and only the first
20 are returned.
You can now use -S prog to use an external program to supply documents to Swish-e. This external program can be used to spider web servers, index databases, or to convert any type of document into html, xml, or text, so it can be indexed by Swish-e. Examples are given in the prog-bin directory.
TranslateCharacters now is done before WordCharacters is checked. For example,
WordCharacters abcdefghijklmnopqrstuvwxyz
TranslateCharacters ñ n
Now El Niño will be indexed as El Nino (el and nino), even though ñ is not listed in WordCharacters.
Previously, stopwords were checked after stemming and soundex conversions, as well as most of the other word checks (WordCharacters, min/max length and so on). This meant that the stopword list probably didn't work as expected when using stemming.
The search parser was rewritten to correct a number of logic errors. Swish-e did not differentiate between meta names, Swish-e operators and search words when parsing the query. This meant, for example, that metanames might be broken up by the WordCharacters setting, and that they could be stemmed.
Swish-e operator characters "*()= can now be searched by escaping with a backslash. For example:
./swish-e -w 'this\=odd\)word'
will end up searching for the word this=odd)word. To search for a backslash character preceed it with a backslash.
Currently, searching for:
./swish-e -w 'this\*'
is the same as a wildcard search. This may be fixed in the future.
Searching for buzzwords with those characters will still require backslashing. This also may change to allow some un-escaped operator characters, but some will always need to be escaped (e.g. the double-quote phrase character).
A bug was fixed in the parse_line() function (in string.c) where backslashes were not escaping the next character. parse_line() is used to parse a string of text into tokens (words). Normally splitting is done at whitespace. You may use quotes (single or double) to define a string (that might include whitespace) as a single parameter. The backslash can also be used to escape the following character when *within* quotes (e.g. to escape an embedded quote character).
ReplaceRules append "foo bar" <- define "foo bar" as a single word
ReplaceRules append "foo\"bar" <- escape the quotes
ReplaceRules append 'foo"bar' <- same thing
user.config file removed.Previous versions of Swish-e included a configuration file called user.config which contained examples of all directives. This has been replaced by a series of example configuration files located in the conf directory. The configuration directives are now described in SWISH-CONFIG.
David Norris has included the files required to build Swish-e under Windows. See src/win32. A self-extracting Windows version is available from the Download page of the swish-e.org web site.
Jean-François Piéronne has provided the files required to build Swish-e under OpenVMS. See src/vms for more information.
Multiple string properties of the same name in a document are now concatenated into one property. A space character is added between the strings if needed. A warning will be generated if multiple numeric or date properties are found in the same document, and the additional properties will be ignored.
Previously, properties of the same name were added to the index, but could not be retrieved.
To do: remove the next pointer, and allow user-defined character to place between properties.
A more general purpose pattern replacement syntax.
Swish-e's XML parser was replaced with James Clark's expat XML parser library.
Swish-e can now use Daniel Veillard's libxml2 library for parsing HTML and XML. This requires installation of the library before building Swish-e. See the INSTALL document for information. libxml2 is not required, but is strongly recommended for parsing HTML over Swish-e's internal HTML parser, and provides more features for both HTML and XML parsing.
Swish-e can be compiled with zlib. This is useful for compressing large properties. Building Swish-e with zlib is stronly recommended if you use its StoreDescription feature.
LST allowed indexing of files that contained multiple documents.
To improve security Swish-e now uses the mkstemp(3) function to create temporary files. Temporary files are used while indexing only. This may result in some portability issues, but the security issues were overriding.
(Currently this does not apply to the -S http indexing method.)
mkstemp opens the temporary with O_EXCL|O_CREAT flags. This prevents overwriting existing files. In addition, the name of the file created is a lot harder to guess by attackers. The temporary file is created with only owner permissions.
Please report any portability issues on the Swish-e discussion list.
Swish-e now uses the environment variables TMPDIR, TMP, and TEMP (in that order) to decide where to write temporary files. The configuration setting of TmpDir will be used if none of the environment variables are set. Swish-e uses the current directory otherwise; there is no default temporary directory.
Since the environment variables override the configuration settings, a warning will be issued if you set TmpDir in the configuration file and there's also an environment variable set.
Temporary files begin with the letters "swtmp" (which can be changed in config.h), followed by two or more letters that indicate the type of temporary file, and some random characters to complete the file name. If indexing is aborted for some reason you may find these temporary files left behind.
Based on Lawrence Philips' Metaphone algorithm, add two new methods of creating a fuzzy index (in addition to Stemming and Soundex).
Changes to Configuration File Directives. Please see SWISH-CONFIG for more info.
The IndexContents directive assigns internal Swish-e document parsers to files based on their file type. The DefaultContents directive assigns a parser to be used on file that are not assigned a parser with IndexContents.
This describes what to do when a meta tag is found in a document that is not listed in the MetaNames directive.
Will ignore text with the listed tags.
Passes words listed to the external Swish-e program when running with -S prog document source method.
Controls parsing and conversion of HTML entities.
The word position is now bumped when a new metatag is found -- this is to prevent phrases from matching across meta tags. This directive will disable this behavior for the listed tags.
This directive works for HTML and XML documents.
This has been changed such that comments are not indexed by default.
The builtin list of stopwords has been removed. Use of the SwishDefault word will generate a warning, and no stop words will be used. You must now specify a list of stopwords, or specify a file of stopwords.
A sample file stopwords.txt has been included in the conf/stopwords directory of the distribution, and can be used by the directive:
IgnoreWords File: /path/to/stopwords.txt
The default is now "yes".
Buzzwords are words that should be indexed as-is, without checking for stopwords, word length, WordCharacters, or any other of the word limiting features. This allows indexing of things like C++ when "+" is not listed in WordCharacters.
Currenly, IgnoreFirstChar and IgnoreLastChar will be stripped before processing Buzzwords.
In the future we may use separate IgnoreFirst/Last settings for buzzwords since, for example, you may wish to index all + within Swish-e words, but strip + from the start/end of Swish-e words, but not from the buzzword C++.
Before Swish-e 2.2 all user-defined document properties were stored in the index as strings. PropertyNamesNumeric and PropertyNamesDate tell it that a property should be stored in binary format. This allows for correct sorting of numeric properties.
Currenly, only integers can be stored, such as a unix timestamp. (Swish-e uses strtoul to convert the number to an unsigned long internally.)
PropertyNamesDate only indicates to Swish-e that a number is a unix timestamp, and to display the property as a formatted time when printing results. Swish does not currently parse date strings; you must provide a unix timestamp.
You may now create alias names for MetaNames. This allow you to map or group multiple names to the same MetaName.
Creates aliases for a PropertyName.
Sets the max length of a text property.
Defines a metaname to use for indexing href links in HTML documents. Available only with libxml2 parser.
Defines a metaname to use for indexing src links in <img> tags. Allow you to search image pathnames within HTML pages. Available only with libxml2 parser.
Allows indexing of image ALT tags. Only available when using the libxml2 parser.
Attempts to convert relative links indexed with HTMLLinksMetaName and ImageLinksMetaName to absolute links. Available only with libxml2 parser.
Allows you to use a regular expression to extract out part of the path of each file and index it with a meta name. For example, this allows searches to be limited to parts of your file tree.
FileMatch is similar to FileRules. Where FileRules is used to exclude files and directoires, FileMatch is used to include files.
Controls which properties are pre-sorted while indexing. All properties are sorted by default.
Sets the level of warning printed when using libxml2.
When using libxml2 to parse HTML, Swish-e will skip files marked as NOINDEX.
<meta name="robots" content="noindex">
Also, comments may be used within HTML and XML source docs to block sections of content from indexing:
<!-- SwishCommand noindex -->
<!-- SwishCommand index -->
and/or these may be used also:
<!-- noindex -->
<!-- index -->
This describes how the content of XML attributes should be indexed, if at all. This is similar to UndefinedMetaTags, but is only for XML attributes and when parsed by libxml2. The default is to not index XML attributes.
XMLClassAttributes can specify a list of attribute names whose content is combined with the element name to form metanames.
If compiled with zlib, Swish-e uses this setting to control the level of compression applied to properties. Properties must be long enough (defined in config.h) to be compressed. Useful for StoreDescription.
Defines a set of characters. If a word is made of of *only* those characters the word will not be indexed.
This configuration directive is used to define the type of "fuzzy" index to create. Currently the options are:
None
Stemming
Soundex
Metaphone
DoubleMetaphone
Changes to command line arguments. See SWISH-RUN for documentation on these switches.
-HControls the level (verbosity) of header information printed with search results.
-xProvides additional header output and allows for a format string to describe what data to print.
-kPrints words stored in the Swish-e index.
-NProvides a way to do incremental indexing by comparing last modification dates. You pass -N a path to a file and only files newer than the last modified date of that file will be indexed.
-D-D no longer dumps the index file data. Use -T instead.
-T-T is used for debugging indexing and searching.
-dNow -d can accept some back-slashed characters to be used as output separators.
-PNow -P sets the phrase delimiter character in searches.
-LSwish-e 2.2 contains an experimental feature to limit results by a range of property values. This behavior of this feature may change in the future.
-vNow the argument -v 0 results in *no* output unless there is an error. This is a bit more handy when indexing with cron.