Search Tools for Web Sites and Intranets

Current Search Tools News

Why the MediaWiki Site Search Stinks

Wikipedia, and particularly the related sites running the software released as MediaWiki, have some of the worst site search I have ever seen. The default installation's query processing is absurdly limited, the retrieval is crippled by bad settings, the relevance is unclear, and the results page is not just ugly but contradictory and confusing. I will be posting more detailed analyses supporting each of these statements. Read more about this analysis or leave a comment - October 2008

6: MediaWiki Highlights the Wrong Text in Search Results

Even worse than the intrusive markup symbols, the MediaWiki Search highlights substrings when the search only retrieves on whole words. It also highlights words that are not indexed and are never part of the search query: those of less than four characters, and words on the very large stopword listing. This is just wrong, and the main Wikipedia has fixed it (recently). However, this wrong default behavior can show up on sites running even the most recent MediaWiki version (1.14a). Read more or leave a comment - Oct. 24, 2008.

5: MediaWiki Markup Symbols in Search Results

MediaWiki's site search does display the search terms matched in the article, with some extracted text from the area around the term, so searchers can understand the context of the match. (For more information, see Matching Search Terms In Context.)

But most search engines remove the page markup (HTML or other) before saving the page, or at least before displaying the results with the match terms in context. MediaWiki Search does not do this, so results will show not just the text from the page, but also hidden text (such as in graphic file names) and markup symbols. Read more or leave a comment - Oct. 21, 2008.

4: MediaWiki Search Results Header is Misleading

The MediaWiki search results page does not actually say how many articles match the search terms. It has confusing and unlabeled number links for results page navigation, and results-per-page setting, right next to each other, but no total for results. Read more or leave a comment - Oct. 17, 2008.

3: MediaWiki Search Has Very Limited Syntax

The default behavior of the MediaWiki search engine is to find only pages which match every word in the search query (find all). When there are quotes around terms in the query, it will only find pages with those terms as a phrase, which is nice. Unlike other search engines, however, there's no way to override this behavior. No options are available to exclude terms from search results, search for several synonyms, automatically use plural word matching, or find substrings using truncation or wildcards. More... or leave a comment - Oct. 16, 2008.

2: MediaWiki Search, Stopword Hell

Stopwords are supposedly words that don't make sense in searches, words that are on many pages and therefore are "noise" in search results. They are generally short, such as a, an, the. However, the MediaWiki search has excludes 547 words as stopwords. But they're perfectly good words, and by ignoring them, the search engine fails in many cases where it should find results. see more details or leave a comment - Oct. 14. 2008.

1: MediaWiki Search Does Not Index Short Words

The MediaWiki search engine defaults to a four-letter minimum word length. Seriously. Not only will it not search for one, two, or three letter words, they're not even in the index: they are completely unfindable. There is no way to search for perfectly reasonable words like fan, lab, qi, or pH. The bartendersdatabase.com wiki can't find rum while Google finds 731 on that same site (though some are duplicates). see more details or leave a comment. - Oct. 13. 2008

Overview Article on Inverted Indexing for Search

This is an expansive and detailed overview, including both practical and theoretical information. The authors report on their own and others tests results finding that inverted indexes are both significantly faster to search and easier to maintain than relational database management systems, signature files and suffix arrays. The article also has a thorough annotated bibliography.

Inverted files for text search engines by Justin Zobel and Alistair Moffat in ACM Computing Surveys. 2006;38(2) (56 pages). Available in PDF format: http://doi.acm.org/10.1145/1132956.1132959 It's $10 if you don't have an ACM account and you have to register on the site. There also seems to be a copy of the PDF file on CiteSeer- Sept. 17. 2008.

Sphinx Search: New SearchTools Report

Sphinx is an open source free search engine, written in C, using both SQL and custom index files to provide a very fast text search. The architecture scales to over a billion records by distributing the index and querying among multiple virtual and real processors. Read more and tell me about your Sphinx experience here. -- July 11, 2008

For more news about search engines, see the News page.

Recent & Interesting Search Links (my Furl archive)