Enterprise Search 101

Avi Rappoport

Search Tools Consulting

About Avi Rappoport

• Avi Rappoport
  • Librarian and long-time search engine analyst
  • Fascinated by user behavior
• Search Tools Consulting
  • Editor & Publisher, SearchTools.com
  • Research, analysis, dissemination
  • Real-world consulting

Defining Enterprise Search

• Web site search
• Corporate / Academic Site
• Online Store
• Information Site
• Directory, Yellow Pages
• Intranet search
• Internal networks
• Crossing departmental lines
• HR and Facilities info
• Research
• Data Silos

Search as an Iceberg

• All people see are the search box and results list
• Invisible
  • Index files
  • Query processing, retrieval, relevance ranking
• Make search a mystery
  • But it's just software

Similarities to Webwide Search

• Techniques and algorithms
• Robot crawlers, HTTP transport
• Very large index files
• Distributed indexing and search processing
• Full-text indexing of unstructured text
• Simple query language
• Relevance ranking of results
• Standard interface conventions

Differences from Web Search

• Limited scope - just a site, set of sites, or intranet
• More control over content
• Few meaningful hyperlinks
• So Page Rank is less useful
• Content in databases, CMSs, etc.
• More tuneable
• Index update scheduling
• Relative value of content for relevance ranking
• No search spam!

Search and Information Architecture

• Defining Information Architecture
• The art and science of organizing information
• Information Architecture enriches search
• Identifies duplicate and missing coverage
• Provides standard vocabulary
• Adds labels for describing documents
• Search supplements IA
• Provide ad-hoc access
• Cross categories
• Supports user vocabularies

Search and Taxonomy Categorization

• Search complements taxonomy, and vice-versa
• Search within a category
  • Improves quality of search results
  • Supplements drill-down links
• Search across categories
  • Avoids forcing choices for ambiguous topics
  • New concepts - may not fit easily in taxonomy
• Taxonomic vocabularies can be confusing
  • Novices and new employees
  • Non-native speakers
  • Search offers alternate access

Search vs. Knowledge Management

• “The process through which organizations generate value from their intellectual and knowledge-based assets”

(CIO Magazine)

• Organizes information, processes and people
• Attempts to regularize implicit knowledge
• Provides collaboration tools

• Search is less ambitious

Simple Search vs. Research

• Text search for most things
• 80% of information needs?
• Short queries
• "Good-enough" answers
• Not so much for research
• Scientific and academic
• Legal discovery
• Medical diagnosis
• Patents and trademarks
• Business and sales information
• Image and multimedia search

Text Search Engines vs. Databases

• Unstructured vs. structured content
• Designed for multiple content sources
• Simple search instead of SQL or other query languages
• Relevance ranking (this is a major issue)
• Response time and database resources
• Special-purpose code is appropriate
• Works in the real world, from eBay to Google

Elements of an Enterprise Search Engine

• Automated tools to collect content
• Indexing text and context
• Multimedia and other complex formats
• Query processing
• Retrieval, recall and precision
• Relevance ranking
• Search results interfaces
• Analytics, metrics and maintenance

Choosing Content To Index

• Information sites: everything!
• Online stores, catalogs
• Product information
• Related information - size charts, return policies
• Intranets
• Intranet portal or core server
• Commonly-used servers
• Create a secondary archive index for weird stuff

Security and Access Control

• Encryption - SSL
  • Search engine uses SSL client
  • Serves results encrypted to avoid
• Basic authentication, single sign-on
• Indexer logs in as a user
• Stores document permissions
• Show inaccessible results as "teasers"
• Warning: can be reverse-engineered
• Collection-level access control
• Hit-level authentication
• Check access status for each result before displaying

Finding Content

• Local servers (web site, intranet, extranet)
• List of "root" URLs
• Otherwise follow links, self-directing
• Other data sources
• File systems & shared servers
• Mailing list archives (index threads), blogs, wikis
• Content Management Systems
• Databases &Legacy programs
• Feeds

How Robot Spiders Work

• Start with a "root" URL (usually home page)
• List and follow all links on that page
• Follow all links from that page
• For each page, repeat
  • Check list to avoid going back to previous pages

Common Problems With Robots

• Pages that are not linked or listed
• Spider disallowed by "robots.txt"
• Problem links
  • JavaScript dynamic links
  • Interactive form links
• Dynamic pages
  • URLs with ? in them
  • May depend on session ID
• Duplicate files
  • Symbolic links, bad redirects, multiple copies of directories
  • Use an algorithm to choose which copy to display

Other Data Sources

• File servers
• Databases
  • Full-text content
  • Product or service information
  • Metadata: library catalog records, yellow pages information
• Information feeds, newswires
• Content Management Systems
• Access options
  • API (Application Programmer Interface)
  • Web Services (SOAP)
  • HTML front end - surprisingly common

File Format Issues

• Primary - HTML & text
• XML
• Text between tags
• Attributes within tags
• Context: store hierarchy structure for text
• Complex file formats
• Flash
• PDF
• Word processing
• Spreadsheets (hard to know what to grab)
• CAD and project files
• Encourage creation of metadata (properties)
• Warning: file formats may change

Indexing - Getting Data

• Extract text
• Text between HTML and XML tags
• Text within special meta tags and properties fields
• Flash: has some tools, limited amount of text
• Ignore formatting for word processing, spreadsheet, slides
• PDF: often a problem with text in order
• Tokenization: separate text into individual words
  • Harder than it looks
  • Various rules for punctuation
  • Asian languages often have no spaces between words

Index & Search Multiple Languages

• Full-text search generally works
• Word matching is simple
• Character sets
• Non-English Roman languages - diacriticals like ß and ç
• Russian, Arabic, Hebrew and Thai
• Chinese, Japanese and Korean even more complex
• Language & charset recognition
• Most documents are not tagged or are not correctly tagged
• Indexers perform language recognition
• Useful for tokenizing, vital for stemming

Index Structures and Content

• Basic Inverted Index
• Alphabetical list of words (not in sentence order)
• Link to source where each word comes from
• Richer index
• Word position in source (for phrase matching)
• Enclosing tag (esp. for metadata)
• Context information
• Anchor text
• Words from links pointing at a file
• Other data structures
• Proprietary improvements on inverted index

Inverted Index Diagram

Indexes: Document Information

• Location (URL / file path / database & record ID)
• External metadata
  • file name
  • size
  • date
  • extension
• Internal metadata
  • title tag or property
  • MIME type
  • date
  • keywords
  • author
  • Dublin Core content

Complex Index Format

Indexing Multimedia

• External metadata
  • Title, topic, keywords
  • May need to extract from link text, surrounding words
• Internal metadata
  • ID3 tags for music
  • Digital photograph information
  • Subtitles (sometimes)
• Graphic text, subtitles - use OCR to convert to digital text
• Speech - use speech-to-text conversion
• Semi-automated systems: best approach
  • Generate indexable data once per file, use human editors
  • Store in metadata repository for later indexing

Indexing: Dates

• Servers lie about dates
  • Frequent wrong settings: 1969, 2040
  • Dynamic pages send the current date
• File systems lie about modification date
  • Store date file was copied
• Applications lie
  • Mark files modified if simply opened and saved
• Indexers do the best they can
  • Checksum to see if file has changed since last index
• CMSs can be saviors
  • Track content modification
  • Enter creation and modification date metadata

Indexing: Stopwords

• Defining stopwords
  • Linguistic (a, an, the)
  • Ubiquitous (company name, "copyright")
• Purpose in not indexing them
  • Reduce size of index files
  • Improve precision, avoid incorrect matches
• Result : failed searches
  • As You Like It
  • The The
  • copyright policy
• Solution
  • Disc space and RAM are cheap now
  • Remove stopwords from query, offer user option to search for them

Indexing: Stemming

• Defining stemming
  • Linguistic truncation
  • Noun pluralization: dishes = dish; goose = geese
  • Verb expansion: run = running = ran
• Purpose in stemming before indexing
  • Reduce size of index files
  • Improve recall: find all forms of a word
• Result - Confused users and weird matches
  • The film Ran is not the same as Run, Lola, Run
• Solution
  • Index exact words
  • Perform stemming at search time
  • Allow users to specify exact matches

The Search Process

Query Processing

• After the user types some words into a search field and clicks the search button...
• Query processing steps
  • Figure out the character set and language
  • Tokenize the text to extract words (same algorithm as the indexer)
  • Check for operators
    • Internet Query Operators: + - "exact match"
    • Boolean Operators: AND OR NOT
    • Others: NEAR, (parentheses)
  • Check for filters and fields
    • title:word inurl:word
    • zone=support

More Query Processing

• Stemming
  • Expand search, at least to plural form of word
• Synonym expansion
  • Limited set of true synonyms
  • dog = canine
  • Automatic teller machine = ATM
• Spellchecking
• Spelling errors, typos, words
• Custom, site-specific dictionary
  • Use search index as spellchecker word list
  • Never suggest anything not on site
  • Offer suggestions for names, acronyms, site-specific terms

Search Retrieval, Recall & Precision

• Retrieval
  • Finding documents matching a particular query
• Recall
  • Finding every relevant document
• Precision
  • Finding only relevant documents
• Must balance: more recall = less precision
  • Show most relevant documents first (precision in display)
  • Understand user needs
    • Casual searches on a big site - more precision
    • Commerce sites - more recall to bring back related results, more chance to sell
    • Legal, medical, academic research - complete recall is valuable

How Retrieval Works

• For every query term
  • Find every document that contains it
  • Store document ID in a list
• Single word queries: done
• Multiple word queries
  • Merge the lists of documents using query operators
    • AND, + operators, find all default: only documents which contain all terms
    • OR operator, find any default: all documents with any term
    • NOT and - operators: documents with the first term but not the next
  • Limit according to fields and zones in query
  • Repeat with all words, operators and limits
• Final list is the search results list (unsorted)

Defining Relevance Ranking

• The likelihood that a document or record will fill an information need
• Term frequency : inverse document frequency (tf:idf)
  • Main algorithm for simple search
  • How many times the query words are in a document
  • How often the word is in the index (rare words are more interesting)
• Other relevance algorithms
  • Vectors and document similarity to query
  • Linguistic analysis and Natural Language Processing
  • Statistical and Bayesian analysis
• Heuristics (rules of thumb)
  • Words matching in special sections: title, metadata
  • Phrase matches for query terms

More on Relevance

• Link analysis less useful in enterprise than on the web
• Page Rank etc. rely on meaningful links
• Intranet and site links tend to be navigational
• Search administrators can define more valuable content
• Product info vs. support notes
• Current vs. old
• Local vs. global
• Weighting rules can reflect that
  • Adjust relevance rank up for valuable pages
  • Reduce rank for others

Visible Search: Search Forms

• Balance simplicity against functionality
• Main form: the search field in the navigation bar
  • Consistent location
  • Longer is better: users worry about short fields

• Simple Search forms - limit options

• Show and change query matching default
  • Any word / All Words / Phrase
• Menu or options for search zones
• Date ranges (if appropriate)
• Do usability tests!

Special Search Interfaces

• Interactive wizards and forms
• Natural language search
• Tried many times -- no good for simple search
  • Delayed gratification rarely rewarded
  • May work for customer service & self-help

Advanced Search

• Even experts prefer simple search
  • Only use advanced features when results are unhelpful
• Most searches are 1-3 words
• Analysis shows only 3-10% of queries use any operators
• Best solution: default to simple search, link to advanced search form
  • A few will appreciate it
  • Those are your power users - librarians, techies

Some Advanced Search Makes Sense

Search Results Page: Overview

• Conform to site design, include navigation
• Search header
  • Results information
  • Search field with current search terms
  • Some options, links to advanced search page
• Results list
  • Relevance-ranked
  • Document or record title, metadata, context, picture...
• Search footer, like the header

Search Results: Visualization

• Fascinating to look at
  • Star charts
  • Topographical displays
  • Interactive flythroughs
  • Hyperbolic trees
  • Look great in demos
• Best for overviews, comprehensive research
  • Finding unexpected synergies
• Simple search does better with simple results
  • Users prefer simple
  • Web search engines have reinforced this
  • Complex results require more time and system resources

Search Results Header

• Search field, with the current query
• Carefully selected search options
  • Match Any Word / All Words / Exact Phase
  • View by date option (if trustworthy)
  • Limit to search zone
• Number of total results found
• Number on this page, and link to next page
• Search Suggestions
• Spelling suggestions

Results Header Example - Good

Search Suggestions

• aka Best Bets, QuickLinks, KeyMatch, Recommendations
• Special-case links for popular searches
  • Top 1% of query words account for 15 to 20% of searches
• Human judgment handles these best
  • Answers seem obvious
  • Requires time and attention, must keep them current
• Creative options
  • Links to landing pages
  • Links to external sites or search engines
  • Link to this search engine with fixed query
• Don't overdo this: rely on the search engine for less common terms

Search Suggestions Example

Result Items Content

• Title of document or name of product
• Location: URL, file path, database record info
• Distinguishing data
  • Page descriptions, snippets of text, pictures, product codes
  • Enough to tell the differences between the items
• Context
  • Match terms with surrounding text
  • Transparency - why did this item get retrieved?

Results Items Example

From New York Magazine

More Information in Results

• Location
  • URL, call number, journal issue and page
• Date - if reliable
• Size - mainly to warn of very large documents
• File type
  • Avoid surprising launches of Acrobat or other application
• Metadata
  • Author, department, brand, product...
• Access status - accessible or protected

Consumer Reports Results Example

Commerce and Catalog Results

• Picture or graphic if possible
• Important attributes
  • Price
  • Color
  • Size
  • Compatibility
  • Availability
• "Buy Now" button
  • Don't require user to go to the product page to buy

Online Store Results Example

Showing Multimedia Results

• Images, video, audio
• Indexing is key
  • Can only search what's available
  • Metadata is vital
• Show thumbnail or keyframe in result
  • Show as much context as possible

Multimedia Results Example

Faceted Metadata Search and Browse

• A powerful interface to semi-structured data
• First adopted by online stores, business directories, library catalogs
• Exposes attributes as part of search results
  • Displays the options relating to search results
  • Dynamically generates categories from metadata
• Far better than form interfaces
• No guessing
• No "dead ends" - only categories with matching metadata are displayed
• "Preview numbers" provide user confidence in clicking

Faceted Metadata: Commerce

Faceted Metadata: Library Catalog

Empty Queries

• Surprisingly common
  • Log analysis may help discover why
• Test default action
• Simplest: show search page
• Javascript to capture search action
  • Do nothing
  • Display an error dialog

Search Failure: No Hits

• Scope: nothing on site or intranet for that topic
• Vocabulary differences
• Restrictive search setting
  • Don't default to matching exact phrase searches
  • Power users may limit search
• Misspellings and typing errors
• Software failure - should never happen

Bad Example of No-Matches Page

Addressing Search Failure

• Design the no-matches page carefully
  • Explain what is and isn't on the site
  • Provide useful navigation links
• Track queries with no matches using log analysis
  • Work with subject specialists to discover user intent
• Add content
  • Common synonyms or non-technical terms in text or metadata
  • Directory page for a topic
    • Description of topic within context of the enterprise
    • Links to related pages
  • Add pages or sections on topics
  • Partner with information provider

Good example of No-Matches Page

Choosing a Search Engine

1. Research the information needs
2. Analyze content on web servers and databases
3. Define scale, platform, system compatibility requirements
4. Compare several likely prospects
   • Indexing tools
   • Retrieval and relevance
   • User-oriented features: spellchecker, synonyms, suggestions
   • Administration tools
   • Special functionality
   • Price

Information Needs Analysis

• What are enterprise-wide or site-wide issues?
• What works already? (don't fix what's not broken)
• Where is the real pain?
  • Systems that drive users nuts
  • Tools that only old-timers understand
  • New content or products
  • Obsolete search engine or database query
  • High-profile users needs
• What requires more complex tools?
  • Scientific research
  • Complex academic research
  • Business intelligence and data mining
  • Don't try to fix these with a simple search engine

Content Survey

• Know what you have
  • Simple static HTML pages
  • Other formats: PDF, text
  • Dynamic systems, CMS publishing
  • Databases and data silos
• Define valuable data
  • Some content may be archival only
• Identify weak areas
  • Recommend content addition
  • Creative work-arounds

Types of Search Engines

• Software Search Engines
  • Controlled by local IT
  • Open-source Search Engines
    • Solid code
    • Generally low on user-oriented features
• Search Appliances
  • Server hardware/software combinations, very simple
  • Issues with development and backup servers
• Remote Search Services (SaaS)
  • Index using robot spiders
  • Queries go to services
  • Results link to original site
  • Low network, hosting, IT load

Testing & Evaluating Search Engines

• Choose two or three good candidates - one is not enough
• Index the content
  • Watch the robot, track errors
  • Test options to index tricky data sources
• Create a test suite (use existing search logs if possible)
  • Both simple and complex queries
  • Spelling, typing and vocabulary errors
  • Force matching issues - many matches, few matches, no matches
  • Save results pages as HTML for later checking
• Analyze the differences among them
  • Variations in indexing
  • Retrieval & response time
  • Relevance Ranking
  • Other elements

Scaling Search

• Check scalable architecture, contact other search engine users
• Move indexing to separate servers
• Distribute index files on multiple servers
  • Could have one index for each initial letter
  • Special indexes for 'stopwords' - a, an, the
  • Merge results
• Distribute search servers, automated rollover
  • Replicate index
  • Accept queries, retrieve and rank results

Search Engine Maintenance

• Index maintenance
  • Make sure obsolete content is removed
  • Check for new content coverage
  • Track technical problems (bad links, servers down)
• Search quality
  • Re-run test suite
  • Compare with original results
  • Add new test queries
• Track user feedback
• Use metrics and log analysis to catch trends

Metrics for Search Engines

• Server uptime, errors
• Index size, number of items, errors
• Search traffic
  • Query with parameters
  • Clicks on results items (if redirect is enabled)
  • Results next-page views

Search Log Analysis

• Most popular query terms
  • Use for search suggestions
  • May indicate site navigation problems
• Query terms not matched
  • Synonym or acronym: add to search suggestions
  • Content on related sites: add to index
  • Related content: add pages
  • Unrelated content: consider partnerships, external links on results pages
• Queries with low click-through
  • Examine results, improve titles or coverage
• Some search engines offer these reports
  • May be very limited in scope
  • Some generate raw logs, can import into database and run reports

Conclusion - Enterprise Search

• Simple search solves most enterprise problems
• Access to local content
• Familiar interface, expectations
• User vocabulary
• Dynamic index of current content
• Not a magic bullet
  • Understand the real information needs
  • Index the right stuff
  • Provide clear and helpful results
  • Maintenance and log analysis pays off
• Questions?

Slides are at: SearchTools.com/slides/ess06/enterprise-search-101.html