Presentation by Search Tools Consulting's Avi Rappoport

Enterprise Search Engines: Critical Success Factors

Avi Rappoport

Search Tools Consulting

KMWorld/Intranets, October 30, 2006

About Avi Rappoport

• Avi Rappoport
  • Librarian and long-time search engine analyst
  • Fascinated by user behavior
• Search Tools Consulting
  • Editor & Publisher, SearchTools.com
  • Research, analysis, dissemination
  • Real-world consulting

Defining “Enterprise Search”

• It's not Web search: Google, Yahoo, Ask, MSN
• Site search
• Corporate / Academic Site
• Online Store
• Information Site
• Directory, Yellow Pages
• Intranet search
• Internal networks
• Crossing departmental lines
• HR and Facilities info
• Research
• Data Silos

Search is Like an Iceberg

• All people see are the search box and results list
• Invisible
  • Index files
  • Query processing, retrieval, relevance ranking
• Search is a mystery
  • But it's just software
  • You can tune it

Similarities to Webwide Search

• Techniques and algorithms
• Robot crawlers, HTTP transport
• Very large index files
• Distributed indexing and search processing
• Full-text indexing of unstructured text
• Simple query language
• Relevance ranking of results
• Standard interface conventions

Differences from Web Search

• Limited scope - just a site, set of sites, or intranet
• More control over content
• Few meaningful hyperlinks
• So Page Rank is less useful
• Content in databases, CMSs, etc.
• More tuneable
• Index update scheduling
• Relative value of content for relevance ranking
• No search spam!

Search and Information Architecture

• Information Architecture
• The art and science of organizing information
• Information Architecture enriches search
• Identifies duplicate and missing coverage
• Provides standard vocabulary
• Adds labels for describing documents
• Search supplements IA
• Provide ad-hoc access
• Cross categories
• Supports user vocabularies

Search and Taxonomy Categorization

• Search complements taxonomy, and vice-versa
• Search within a category
  • Improves quality of search results
  • Supplements drill-down links
• Search across categories
  • Avoids forcing choices for ambiguous topics
  • New concepts - may not fit easily in taxonomy
• Taxonomic vocabularies can be confusing
  • Novices and new employees
  • Non-native speakers
  • Search offers alternate access

Search and Knowledge Management

• “The process through which organizations generate value from their intellectual
    and knowledge-based assets”
                                                                 -- CIO Magazine
  
  
• Organizes information, processes and people
• Attempts to regularize implicit knowledge
• Provides collaboration tools

• Search is less ambitious

Simple Search vs. Research

• Text search for most things
• 80% of information needs?
• Short queries
• "Good-enough" answers
• Not so much for research
• Scientific and academic
• Legal discovery
• Medical diagnosis
• Patents and trademarks
• Business and sales information
• Image and multimedia search

Text Search Engines vs. Databases

• Unstructured vs. structured content
• Text search is designed for multiple content sources
• Simple search query language, instead of SQL
• Relevance ranking (this is a major issue)
• Database resources load can be heavy
• Quick response is vital
• In this case, special-purpose code is appropriate
• Text search in the real world used by high-demand sites from eBay to the IRS

Elements of a Search Engine

• Automated tools to collect content
• Indexing text and context
• Multimedia and other complex formats
• Query processing
• Retrieval, recall and precision
• Relevance ranking
• Search results interfaces
• Analytics, metrics and maintenance

Defining Content To Be Searched

• General principal: index unless there's a reason not to
  • Search gets blamed when people can't find stuff
  • No one can anticipate user needs
  • Content found won't be duplicated
• Create a Content Inventory
  • Work with the Intranet or portal team, IA, and other groups
  • Store information about servers, hosts, directories and pages
  • Track changes
  • Avoid excess processing
• Start with Intranet portal or core server
  • Then go on to commonly-used servers
  • Create a secondary archive index for weird stuff

Finding Content

• Local servers (web site, intranet, extranet)
• List of "root" URLs
• Otherwise follow links, self-directing
• Other data sources
• File systems & shared servers
• Mailing list archives (index threads)
• Blogs
• Wikis
• Content Management Systems
• Databases &Legacy programs
• Feeds

How Robot Spiders Work

• Start with a "root" URL (usually home page)
• List and follow all links on that page
• Follow all links from that page
• For each page, repeat
  • Check the URL list to avoid going back to previous pages
• May discover previously unknown servers

Common Problems With Robots

• Pages that are not linked or listed - can add to starting list
• Spider disallowed by "robots.txt"
• Crashing fragile web servers
  • Limit robot spider speed at first
  • Note problems in Content Inventory
• Problem links
  • JavaScript menus
  • Interactive form links
• Dynamic pages
  • May depend on session ID or cookie
• Duplicate files
  • Symbolic links, bad redirects, multiple copies of directories
  • Solutions
    • Remove duplicate directory from index
    • Set rules for which duplicate to index
    • Index everything and just hide duplicates

Infinite Links and Black Holes

• Infinite links
  • Dynamic URLs that are automatically generated
  • Example: calendar programs that always have a link to the next year.
  • Set indexing rule to avoid the whole problem area
  • Set indexing rule to stop indexing after a certain number of links from a root.
• Black holes
  • Misconfigured links that go on forever
  • Example: a 404 "page-not-found" that links to a subdirectory of itself
    • example.com/error/error/error/
  • Keep following it and the robot might crash the server
  • Set indexing rules on the depth (number of subdirectories) in indexing paths

Other Data Sources

• File servers
• Email archives
• Databases
  • Full-text content
  • Product or service information
  • Metadata: library catalog records, yellow pages information
• Information feeds, newswires
• CMS: Content Management Systems
• Access options
  • API (Application Programmer Interface)
  • Web Services (SOAP)
  • HTML front end - surprisingly common

Access Control and Authentication

• Work with security teams to create a search access policy
• Simplest: index everything, show all results
  • Access control kicks in when user clicks result link
  • Good for teasers and mildly private data
• Warning: can be reverse-engineered
• Basic authentication (user name / password, htaccess)
• Indexer logs in as a user
• Stores document permissions
• Collection-level access control
• Hit-level authentication
• Check access status for each result before displaying
• Single sign-on integration is great

Indexing - Extracting Words in Context

• Index the full text: every word of every item
• Tokenization: separate text into individual words
  • Harder than it looks
  • Language-based rules for punctuation
  • Asian languages often have no spaces between words
• Structure: basic inverted index is very efficient
  • Alphabetical list of words (not in sentence order)
  • Link to source where each word comes from
• Rich Index: more than just a bag of words
• Index structure extensions and proprietary formats
  • Can add valuable features

Inverted Index Diagram

Indexing Simple File Formats

• Plain text files
  • Words and position
• HTML files
  • Body text between tags
  • Parent tag - some indicate importance
    • <h1>
    • <b>
  • Tag attributes
    • img tag alt attribute - may indicate content of image
• XML
  • Define item start and stop
    • Many XML files contain multiple items
• Semantic hierarchy tags
  • May be valuable context
  • Example: meeting minutes, tagging the speakers

Indexing Complex File Formats

• Flash: limited amount of text
• PDF: often a problem with text in order
• Word processing files
  • Extract all visible text, ignore layout
  • Try to ignore hidden text (MS Word)
• Spreadsheets (hard to know what to grab)
• CAD and project files
• Encourage creation of metadata (properties)
• Warning: file formats will change

Indexing Database Records

• Consider the nature of the content and searching
  • Single product or item may have complex aspects
  • Example: shirts come in many colors and sizes
  • Best to combine aspects into a core item for indexing
• Index all text as part of that item
  • Crosses the barriers of database fields
• Mark text with the field names
  • Treat field names like the meta tags

Indexing Multimedia

• External metadata
  • Title, topic, keywords
  • May need to extract from link text, surrounding words
• Internal metadata
  • ID3 tags for music
  • Digital photograph information
  • Subtitles (sometimes)
• Graphic text, subtitles - use OCR to convert to digital text
• Speech - use speech-to-text conversion
• Semi-automated systems: best approach
  • Generate indexable data once per file, use human editors
  • Store in Content Inventory for later indexing

Indexing - Languages/Charsets

• Full-text index and search often works without special programming
• Word matching is relatively simple
• Character sets for tokenization
• Non-English Roman languages - diacriticals like ß and ç
• Russian, Arabic, Hebrew and Thai
• Chinese, Japanese and Korean even more complex
• Language-specific issues
  • Spelling suggestions
  • Stemming
• Language & charset recognition
• Most documents are not tagged or are not correctly tagged
• Best to do recognition once, and store in the Content Inventory

Indexing - Dealing With Dates

• Servers lie about dates
  • Frequent wrong settings: 1969, 2040
  • Dynamic pages send the current date
• File systems lie about modification date
  • Date file was copied, no content change
• Applications lie
  • Mark files modified if simply opened and saved
• Indexers do the best they can
  • Checksum to see if file has changed since last index
• Content Inventory is the place to store date info

Indexing - Yes, Even Stopwords

• Defining stopwords
  • Linguistic (a, an, the)
  • Ubiquitous (company name, "copyright")
• Purpose in not indexing them
  • Reduce size of index files
  • Improve precision, avoid incorrect matches
• Result : failed searches
  • As You Like It
  • The The
  • copyright policy
• Solution
  • Disc space and RAM are cheap now
  • Remove stopwords from query, offer user option to search for them

Indexing - Stem Later

• Stemming
  • Linguistic truncation
  • Noun pluralization: dishes = dish; goose = geese
  • Verb expansion: run = running = ran
• Purpose in stemming during indexing
  • Reduce size of index files
  • Improve recall: find all forms of a word
• Result - Confused users and weird matches
  • The film Ran is not the same as Run, Lola, Run
• Solution
  • Index exact words
  • Perform stemming at search time
  • Allow users to specify exact matches

Index Document Information

• Location (URL / file path / database & record ID)
• External metadata
  • file name
  • size
  • date
  • extension
• Internal metadata
  • title tag or property
  • MIME type
  • date
  • keywords
  • author
  • Dublin Core content

Complex Index Format

Break

• Questions about indexing?

The Search Process

Query Processing

• After the user types some words into a search field and clicks the search button...
• Query Processing steps
  • Figure out the character set and language
  • Tokenize the text to extract words (same algorithm as the indexer)
  • Check for operators
    • Internet Query Operators: + - "exact match"
    • Boolean Operators: AND OR NOT
    • Others: NEAR, (parentheses)
  • Check for filters and fields
    • title:word inurl:word
    • zone=support

More Query Processing

• Stemming
  • Expand search, at least to plural form of word
• Spellchecking - now expected
• Spelling errors, typos, words
• Custom, site-specific dictionary
  • Use search index as spellchecker word list
  • Only suggest words with at least one match
  • Never suggest anything not on site
• Synonym expansion
  • Small set of true synonyms in most cases
    • dog = canine
  • Context-based expansions
    • ATM = Automatic teller machine (bank site)
    • ATM = Asynchronous Transfer Mode (telecom site)
  • Not clear how to present this - test the UI

Search Retrieval, Recall & Precision

• Retrieval
  • Finding documents matching a particular query
• Recall
  • Finding every relevant document
• Precision
  • Finding only relevant documents
• Must balance: more recall = less precision
  • Show most relevant documents first (precision in display)
  • Understand user needs
    • Casual searches on a big site - more precision
    • Commerce sites - more recall to bring back related results, more chance to sell
    • Legal, medical, academic research - complete recall is valuable

How Retrieval Works

• For every query term
  • Find every document that contains it
  • Store the document ID in a list
• Single word queries: done
• Multiple word queries
  • Merge the lists of documents using query operators
    • AND, + operators, find all default: only documents which contain all terms
    • OR operator, find any default: all documents with any term
    • NOT and - operators: documents with the first term but not the next
  • Limit according to fields and zones in query
  • Repeat with all words, operators and limits
• Final list is the search results list (unsorted)

Defining Relevance Ranking

• The likelihood that a document or record will fill an information need
• Term frequency : inverse document frequency (tf:idf)
  • Main algorithm for simple search
  • How many times the query words are in a document
  • How often the word is in the index (rare words are more interesting)
• Other relevance algorithms
  • Vectors and document similarity to query
  • Linguistic analysis and Natural Language Processing
  • Statistical and Bayesian analysis
• Heuristics (rules of thumb)
  • Words matching in special sections: title, metadata
  • Phrase matches for query terms

More on Relevance

• Relevance is situational, not an abstract ideal
  • Even for the same person in the same search session
  • Understanding changes
• Link analysis less useful in enterprise than on the web
• Page Rank etc. rely on meaningful links
• Intranet and site links tend to be navigational
• Intranet: some rules for likely valuable content
• Product info vs. support notes
• Current vs. old
• Local vs. global
• Weighting rules can reflect that
  • Adjust relevance rank up for widely-useful pages
  • Reduce rank for others

Break

• Questions on query processing, retrieval or relevance?

Visible Search: Forms and Results Pages

• Search forms and fields
  • Include a search field in the navigation bar
• Offer local search option for intranet hosts
• Balance simplicity against functionality
• Advanced search sometimes works
• Results pages
  • Page design and navigation
  • Search-specific elements
  • Search result items
• No-match pages
  • How to help your users

Visible Search: Search Forms

• Search field in the navigation bar
  • Consistent location
  • Longer is better: short fields lead to short queries

• Simple Search forms - limit options
• Show and change query matching default
  • Any word / All Words / Phrase
• Menu or options for search zones
• Date ranges (if appropriate)
• Do usability tests!

Special Search Interfaces

• Interactive wizards and forms
• Natural language search
• Tried many times -- no good for simple search
  • Delayed gratification rarely rewarded
  • May work for customer service & self-help

Advanced Search is A Tricky Problem

• Most searches are 1-3 words
• Only 3-10% of queries use any operators
• Even experts start with simple searches
• Only some searches need advanced tools
• Best solution: default to simple search, link to advanced search form
  • A few will appreciate it
  • Those are your power users - librarians, techies
  • Frequent users with complex requirements (eBay)

Some Systems Need Advanced Search

Ebay users have specific requirements and are willing to use the advanced search form.

Likewise, scientific, technical and business intelligence researchers may want to use advanced search for specific topics.

Search Results Page: Overview

• Conform to site design, include navigation
• Search header
  • Results information
  • Search field with current search terms
  • Some options, links to advanced search page
• Results list
  • Relevance-ranked
  • Document or record title, metadata, context, picture, more...
• Search footer, like the header

Visualization in Search Results

• Beyond the simple item list
  • Star charts
  • Topographical displays
  • Interactive flythroughs
  • Hyperbolic trees
  • Look great in demos
• Best for overviews, comprehensive research
  • Finding unexpected synergies
• Simple search does better with simple results
  • Users prefer simple
  • Web search engines have reinforced this
  • Complex results require more time and resources

Search Results Header

• Show search field with the current query
• Carefully selected search options
  • Match Any Word / All Words / Exact Phase
  • View by date option (if trustworthy)
  • Limit to search zone
• Number of total results found
• Number on this page, and link to next page
• Search Suggestions
• Spelling suggestions

Results Header Example

Search Suggestions

aka Best Bets, QuickLinks, KeyMatch, Recommendations

• Special-case links for the most popular searches
• Top 1% of query words account for 15 to 20% of searches
• Use search logs instead of guessing popularity
• Human judgment handles these best
  • Answers seem obvious
  • Requires time and attention, must keep them current
• Creative options
  • Links to landing pages
  • Links to external sites or search engines
  • Link to this search engine with fixed query
• Don't overdo this
  • Rely on the search engine for less common term

Search Suggestions Example

Basic Result Items Content

• Title of document or name of product
• Location: URL, file path, database record info
• Distinguishing data
  • Page descriptions, snippets of text
  • Pictures - often show vital distinctions
  • Product codes or version numbers
  • Enough to tell the differences between the items
• Context
  • Match terms with surrounding text
  • Transparency - why did this item get retrieved?

Simple Results Items Example

More Information in Results

• Location
  • URL, call number, journal issue and page
• Date - if reliable
• Size - mainly to warn of very large documents
• File type
  • Avoid surprising launches of Acrobat or other application
• Metadata
  • Author, department, brand, product...
• Access status - accessible or protected

Consumer Reports Results Example

Commerce and Catalog Results

• Picture or graphic if possible
• Important attributes
  • Price
  • Color
  • Size
  • Compatibility
  • Availability
• "Buy Now" button
  • Don't require user to go to the product page to buy

Online Store Results Example

Searching Multimedia Content

• Images, video, audio
• Indexing is key
  • Can only search what's available
  • Store metadata for indexing
• Show thumbnail or keyframe in result
  • Show as much context as possible

Multimedia Results Example

Faceted Metadata Search and Browse

• A powerful interface for structured and semi-structured data
• First adopted by online stores, business directories, library catalogs
• Exposes attributes as part of search results
  • Displays the options relating to search results
  • Dynamically generates categories from metadata
• Far better than form interfaces
• No guessing
• No dead ends - only categories with matching metadata are displayed
• "Preview numbers" provide user confidence in clicking

Faceted Metadata: Commerce

Faceted Metadata: Library Catalog

Handling Empty Queries

• Surprisingly common
  • Log analysis may help discover why
• Test default action
• Simplest: show search page
• Javascript to capture search action
  • Do nothing
  • Display an error dialog

Why Some Queries Have No Matches

• Misspellings and typing errors
• Vocabulary differences
  • coach vs. bus
  • Common problem for new hires
• Scope: nothing in the index for that topic
  • Confusion between intranet search and webwide search
  • Searching in wrong zone
• Content is missing
• Too-restrictive search setting
  • Default set to matching exact phrase searches
  • Power users may limit search
• Software or hardware failure

What To Do for No-Matches Queries

• Design the no-matches page carefully
  • Explain what is and isn't on the site
  • Provide useful navigation links
• Track no-match queries by using log analysis
  • Work with subject specialists to discover user intent
• Implement a spellchecker
• Add search suggestions for the most common
• Add content
  • Common synonyms or non-technical terms in text or metadata
  • Add pages or sections on desired topics
  • Landing page for a topic or external application
    • Description of topic and links to related pages
  • Partner with information providers to extend content

Bad Example of a No-Matches Page

Good Example of a No-Matches Page

Break

• Questions about Search Interfaces, Results Pages, Facets, Search Failure?

Choosing a Search Engine

1. Research the information needs
2. Analyze content on web servers and databases
3. Define scale, platform, system compatibility requirements
4. Compare several likely prospects
• Indexing tools
• Query processing: spellchecker, synonyms, suggestions
• Retrieval and relevance
• Administration tools
• Special functionality
• Price
5. Implement and test test test!
• Most search engine developers don't test user experience at all

Content Survey

• Work with intranet team, IA, taxonomists
• Know what you have
  • Simple static HTML pages
  • Other formats: PDF, text
  • Dynamic systems, CMS publishing
  • Databases and data silos
• Define valuable data
  • Some content may be archival only
• Identify weak areas
  • Recommend content addition
  • Creative work-arounds

Types of Search Engines

• Software Search Engines
  • Supported by local IT
  • Easy to install development, staging, rollover servers
  • Commercial: wide range of prices and features
  • Open-Source: low on user-oriented features
• Search Appliances
  • Server hardware/software combinations, very simple
  • Look at licensing time limits and additional servers
• Remote Search Services (SaaS)
  • Index using robot spiders
  • Queries go to services
  • Results link to original site
  • Low network, hosting, IT load

Evaluating Search Engines - I

• Choose two or three good candidates - one is not enough
  • You will learn a lot from the simple act of installing
• Index some representative servers
  • Set a starting page and define limits
  • Be sure to control crawling speed
    • Rickety servers may fail when every page is indexed
  • Find ways to avoid indexing navigation text
• Start evaluating ways to index complex data sources
  • Connect to source API or service
  • Consider installing an HTML front-end to the data
• Compare lists of content indexed
• Compare content of pages indexed

Evaluating Search Engines - II

• Create a test suite (use existing search logs if possible)
  • Simple and complex queries, spelling, typing and vocabulary errors
  • Force matching issues
    • Many matches - e.g. name of product or company
    • Few matches - technical terms
    • No matches - force with nonsense such as "zxcvbnm"
  • Save results pages as HTML for later checking
• Analyze the differences among them
  • Retrieval & response time
  • Relevance ranking
• Look at reporting and analytics
• Admin features will be clear by this time

Think About Scaling Search

• Many search engines handle a million records with ease
• Traffic is less of a concern: intranets are light usage
• Analyze search engine architecture
  • Talk to search admins with larger systems
• Common methods of scaling
  • Move indexing to separate servers
  • Distribute index files on multiple servers
    • Could have one index for each initial letter
    • Special indexes for 'stopwords' - a, an, the
    • Merge results
  • Distribute search servers, automated rollover
    • Replicate index
    • Accept queries, retrieve and rank results

Search Engine Maintenance

• Index maintenance
  • Make sure obsolete content is removed
  • Check for new content coverage
  • Track technical problems (bad links, servers down)
• Search quality
  • Re-run test suite
  • Compare with original results
  • Add new test queries

Metrics for Search Engines

• Server uptime, errors
• Index size, number of items, errors
• Search traffic
  • Queries
  • No-matches queries
  • Empty queries
  • Clicks on results items (if the server supports this feature)
  • Results next-page views

Search Log Analysis

• Most popular query terms
  • Use for search suggestions
  • May indicate site navigation problems
• Query terms not matched
  • Synonym or acronym: add to search suggestions
  • Content on related sites: add to index
  • Related content: add pages
  • Unrelated content: consider partnerships, external links on results pages
• Queries with very high or very low click-through
  • Examine results, improve titles or coverage
• Some search engines offer these reports
  • May be very limited in scope
  • Some generate raw logs, can import into database and run reports

Conclusion - Enterprise Search Success Factors

• Simple search solves most enterprise problems
• Access to local content
• Familiar interface, expectations
• User vocabulary
• Dynamic index of current content
• Not a turnkey system though
  • Understand the real information needs
  • Index the right stuff
  • Provide clear and helpful results
  • Maintenance and log analysis pays off
• Questions?