Annotated links to books, reports and articles which provide general information about web site, intranet and portal search engines, including how to evaluate and choose software. See also Background Information listings, Links, Training, Newsgroups & Mailing Lists, Comparative Reviews, and Tools Listings.

For a local overview, see the Guide to Search Tools section.

• Tools For Web Searches Get A New Focus InternetWeek, December 6, 1999 by Noah Shachtman
  Describes how e-commerce sites are using customized versions of AskJeeves, Copernic, Google and NorthernLight to improve their response to customer service and product questions.
  
  
• Robots and Spiders and Crawlers Inktomi White Paper, September 1999 by Avi Rappoport
  Detailed discussion, of how search engine indexing robots follow links and read Web pages to store the information in search indexes. Includes coverage of problem areas such as image maps, frames, JavaScript and dynamic data. Notes describe how the Inktomi Search Software Spider handles these problems.
  
  
• (PDF file, 48K) A Method for Intranet Search Engine Evaluations by Dick Stenmark, in Proceedings of IRIS22 Kþkšlþ, T. (Ed.), Keuruu, Finland, August 7-10, 1999.
  A thoughtful and practical approach to evaluating server search engines, grouping functions and weighting according to importance. The methodology provides checklists of important issues and was tested in several corporate situations.
  
  
• Searching Your Site - HTML About.com, July 13, 1999 by Jennifer Kyrnin
  Short description of site search options, including short descriptions of selected remote search services, Perl scripts and Javascript engines.
  

• Modern Information Retrieval Ricardo Baeza-Yates & Berthier Ribero-Neto, Addison-Wesley Longman, May 1999, ISBN 020139829x, $50
  Covers the current state of information retrieval, including changes brought by the Web to a field that was previously oriented towards academia, libraries and corporate networks. Support the SearchTools site: buy the book from Amazon or FatBrain.com and we'll get an affiliate fee.
  
  
• User Interfaces and Visualization in Modern Information Retrieval chapter by Marti Hearst
  Valuable academic study of interfaces for information retrieval and searching, including graphical overviews and visualization. This is a chapter of the Modern Information Retrieval book described above.

• Managing Gigabytes: Compressing and Indexing Documents and Images (2nd edition) by Ian H. Witten, Alistair Moffat, and Timothy C. Bell. Morgan-Kaufmann Publishers; April 1999; ISBN 1558605703, $54.95
  Good coverage of issues of very large document collections, including compression, indexing and querying options. Praised by Steve Kirsch of Infoseek, among others. See also the MG web site and the excellent reviews. If you buy it from Amazon or FatBrain.com using these links, you will support the SearchTools site.
  
  
• The Ultimate Search: How to Evaluate Search Servers Intraware NetInsights, December 10, 1998 by Molly Bellero
  Covers indexing capacity and options, search features, administration, and interface customization.

• Add search to your site CNET Builder.com, November 17, 1998 by Avi Rappoport
  A comprehensive set of instructions including background information, choosing the right site search engine for your site, testing and installing the software, designing the interface, and setting up a maintenance program. Includes many links to sites and products, summaries of some of the most attractive programs and an example of the installation process using the DNA Files web site.
  
  
• Perfecting Corporate Search Engines Inter@ctive Week, September 10, 1998 by Steven Vonder Haar
  Describes the problems with finding information in corporate intranets, strategies by companies and vendors to improve the situation. Options include more training, concept mapping approaches and directory cataloging (like Yahoo).
  
  
• Search Engines for Intranets Information Today, July 1998 by Nina Platt
  Extremely thoughtful overview including the "right questions" to ask when choosing a search tool for an intranet, how to develop an implementation plan, and a table of comparisons among QuickFinder (Novell intraNetWare), Microsoft Index Server and Netscape Compass Server.
  
  
• Mac servers get more search tools MacWEEK, July 13, 1998 by Avi Rappoport
  Description of the benefits and issues of site search tools, including indexers vs. crawlers. Covers the features of iHound, Boolean Search, Phantom and WebSTAR Search.
  
  
• In pursuit of intelligent search engines Datamation, May 1998 by Dan Orzech
  Describes problems with Intranet search engines, which do well with matching words but less well with interpreting relevance. Approaches include Natural Language Processing, thesaurus use, automated classification, personalized "push" search results, going beyond HTML to databases and other file formats, image matching, graphical interfaces and visualization, and falling prices.
  
  
• Search Engines for Local Web Sites March 26, 1998 by Judy Brown
  Covers the history of Web search engines, how indexing robots approach pages, subject directories (also called categories), defining site needs, dealing with legacy data, background on evaluating search engines, listings of various options including platform and file format compatibility, and thoughts for the future.
  
  
• Search for the stars? Looking for sites with exceptional search engines Computerworld, March 9, 1998 by Beth Roed and Jared O'Connor
  Reviews of 30 web site search engines in 10 industries: Computer Hardware, Computer Software, Telecommunications, Retail, Consumer Goods, General Manufacturing, Financial Planning/Insurance, Newspaper, Government, Education. Features commended including ease of use, speed, good recall and precision, helpful summaries, clear presentation of advanced features such as filters, results sorting, "area match" and organization (information architecture), targeted features (price-range searching at Wal-Mart), user tips, tolerance of spelling errors.
  
  
• Putting search on your site Computerworld, March 9, 1998 by Kevin Burden
  Helpful information, including the amount of time required for maintenance and administration, formats supported, data location, user requirements, relevancy and performance.
  
  
• Designing Your Search Interface Web Review March 6, 1998, by Peter Morville and Lou Rosenfeld
  Excerpts from the excellent book Information Architecture for the World Wide Web describe how to support different modes of searching, integrate search with browsing, conform to your site's design, and clarify search options. You can buy the book online from Amazon or Fatbrain online bookstore, and support this site -- we appreciate that.
  
  
• Fine-tuned search engines LANTimes, March 2, 1998 by Emily Kay
  Good overview of site searching in corporate Intranet sites. Describes why some sites need more than traditional Boolean searching, multimedia file formats, and canned searches, featuring BRS/Search (Dataware) products. Covers performance and hardware capacity issues.
  
  
• Why On-Site Searching Stinks Eye for Design, 1998 by Jared Spool
  Provocative article shows several examples of bad search interface design. Includes examples of problems with confusing substring searching, having search help on a separate page, misspellings, unclear results, confusing page titles in results listings, and a lack of indexing which finds pages that match the words but not the meaning of the queries.
  
  
• Web Developer.Com Guide To Search Engines Wes Sonnenreich and Tim MacInta: John Wiley & Sons, February 1998, ISBN 0471246387 $34.99.
  A wide-ranging book covering everything from the beginnings of the robot spiders crawling and indexing the web to analysis of the major webwide search engines to detailed information on installing and configuring six local site search tools. The programs covered are AltaVista Search Intranet, Excite for Web Servers, Harvest, ht://Dig, Phantom and Ultraseek (now Inktomi Search Software). Also describes BDDBot: An ongoing collaborative project, to create a Java web server and search spider, using open source under the GNU public license. Use these links to buy from Amazon or Fatbrain and you'll support this site.
  
  
• Casting an Information Net Upside, February 02, 1998, by Jeff Ubois
  Overviews of Site Search Tools from an industry point of view, includes information on search engines, new forms of visualization, collaborative filters, enterprise use of these tools, and corporate info.
  
  
• Indexing the Video Frontier Wired News, February 5, 1998 by Chris Oakes
  Describes the issues of indexing and searching video data, which is much harder than text. Algorithms must find meta patterns, such as scene changes, to break down the data into manageable parts -- and these are different in different genres of video, such as documentaries or sports. Products make good use of closed-captioned text, text within the video and analysis of audio tracks to create storyboards and previews. More subtle indexing will require image analysis and pattern recognition. Products also work with Microsoft's NetShow streaming audio and video products. Includes information about Excalibur, Virage and Magnifi.
  
  
• Seek And Ye Shall Be Found -- Or Else Web Developer: Undated (web article), checked 1998/03/30, by Eric C. Richardson
  Why and how to choose a site search tool, covering both commercial and free/shareware options.
  
  
• Choosing and Using a Site Search Engine Web Review, Nov. 21, 1997, by Daniel A. Tauber and Brenda Kienan
  Good article on the process of selecting and implementing site search tools, aimed mainly at Unix webmasters. Includes a table of features for various tools.
  
  
• Search for differentiation -- Text-search software vendors are still looking for the market's sweet spot, but profits are elusive Information Week, October 6, 1997 by Justin Hibbard.
  Describes the search-and-retrieval industry status, including the extension towards knowledge management.
  
  
• Retooling for intranet: PC Week Labs tells how vendors revamp search engines PCWEEK, August 11, 1997 by Jim Rapoza
  Describes web site search engine products which are aimed at intranets rather than public sites using search agents, expanded capacity and web publishing of legacy data. Covers Netscape Compass Server 3.0, Verity IntelliServ, AltaVista Search Intranet Extension '97 and Odyssey Isys:web.
  
  
• In search of .. IT managers warm slowly to search technology for their intranets. PC WEEK: (August 11, 1997)
  
  
• Search engines: An intranet users best friend Infoworld Electric, August 11, 1997, by Sari Kalin
  Describes trends in search engine technology such as indexing realtime data, including news feeds and email; visualization tools for clustering and explaining data.
  
  
• Search Usability: Search and You May Find Alertbox column, July 16, 1997 by Jakob Nielsen.
  Very helpful results of research on web site usability finds that half of all users are "search-dominant" (they use the search field as soon as they can). Recommendations are to put a search button on every page, to index the entire site rather than selected documents, and to avoid requiring Boolean operators in the default search .
  
  
• Search Engine Technologies for the World Wide Web and Intranets Bohdan O. Szuprowicz: Computer Technology Research, June 1997, ISBN: 1566079934 $55.01
  Looks promising but is currently out of print and unavailable.
  
  
• Searching for Answers WinMag.com, July 1997 by Tom Henderson
  Overview of how indexing and searching work, Oracle's ConText and SQL issues, inverted indexing, Verity Search '97 and Microsoft Index Server.
  
  

• Site Search and Retrieval Tools WebServer Online, June 1997 by Lang Zerner
  Good technical discussion of indexes, site requirements, hardware and resources, extra features, and search glossary. Includes a somewhat dated buying guide.
  

• Sorting Through Search Engines Web Techniques: June, 1997, by Alex Lange
  Good article covering the basic issues of choosing a site search tool, chart describing both commercial and free/shareware options.
  
  
• The Searchable Kingdom BYTE Magazine, June, 1997, Edmund X. DeJesus
  Good evaluations of the technical underpinnings of the indexing and search tools for intranets. Covers Verity Knowledge Base Network, Open Text LiveLink, Frontier Technologies CyberSearch, Infoseek Ultraseek [bought by Inktomi, June 2000 -ed.], Inktomi's HotBot, Lycos/Inmagic, AltaVista Intranet.
  
  
• Information Retrieval for Your Intranet: Tools to make corporate data more accessible PC Magazine, May 6, 1997 by Scot Finnie
  Helpful discussion of the issues in setting up an intranet search system, including search techniques, indexing, operating systems, crawling, data types, database access, images, results formatting, concept clustering and agents. Provides practical tips on how to choose a system.
  
  
• Clarifying Search: A User-Interface Framework for Text Searches D-Lib Magazine, January 1997 by Ben Shneiderman, Don Byrd and W. Bruce Croft
  Ideas about designing search interfaces, with a "framework" of four steps: formulation (defining the searchable data and search options), action (issuing the search), review of results (results formats) and refinement (adding relevance feedback). Also describes how principles of user interface apply to search interfaces and includes a case study of redesigning the search interface to the Congressional Record.
  
  
• Add a search engine to your site Chami.com, December 12, 1996
  Slightly elderly but still useful checklist of the various options for site search including integrated server search, separate applications, CGI scripts, applets and subsets of webwide search engines.
  
  
• A review of robot based Internet search services Report for Project Desire, August 1996 by T. Koch, A. Ardö, A. Brümmer and S. Lundberg
  Comprehensive coverage of both webwide and local site searching with robot spiders as of 1996. See in particular section 4.2, Analysis of full text retrieval systems and Annex 3: Analysis of fulltext retrieval software.
  
  
• The PIPER Letter: Web Site Search Engine November, 1996
  Good Overviews with useful descriptions of the technical issues of site search tools.
  
  
• WebMaster's Technology Notes - Search
  Elderly (dated 1996) but thoughtful commentary on site search technology.