Organizing Information
for Better Search Results

Avi Rappoport

Search Tools Consulting

About Me

• Librarian, software developer
• Editor of SearchTools.com since 1998
• Independent analyst
• Enterprise and site search consulting

Enterprise Search Goals

• Think of the big picture
• Consider what kinds of information you have now
• Learn about true information needs
  • Don’t assume it’s all about research
  • Don’t trust what people think they need
  • Do ask what people do and what they look for
  • Do find bottlenecks and problems
• Personas and scenarios
• Each employee has several roles
• Identify the common needs

Integrate Information Architecture

• Ride on IA’s heavy lifting
• Give extra weight to landing pages
• Add vocabulary to search suggestions
• Identify document types
• Work with taxonomy and hierarchy
• Treat category names as external metadata
• Show search results with category context
• Provide search within categories

Finding Content

• Use a robot spider to crawl the internal servers
  • Spidering problems with JavaScript
  • Indexing problems with graphic text
• Index blogs and wikis
  • One topic on one page version
• Index mailing list archives and email public folders
• Databases and "silos"
• Usually have some form of HTML front end
• Generate list of links to all records
• Web services
• APIs (Application Programmer Interfaces)

Indexing Coverage

• Comprehensive coverage - user expectations
• ROT - Redundant, Outdated and Trivial
• Redundant
  • Duplicate servers, directories or documents
  • Good search engines "deduplicate" documents
  • Multiple versions of documents - rules for deletion
• Obsolete
• Hard to tell what will be valuable
• Best solution: create a separate archive search
• Trivial
• Document your definitions of trivial

Other Content Issues

• Character set issues
  • If character set is not explicit, must deduce it
  • Indexing into Unicode is best solution
• Languages
  • Important for later pluralized word matching
• Missing metadata
  • Unique and descriptive titles are very important
• Date problems
  • True content modification dates are valuable
  • Servers can be set wrong -- far past or future
  • Some dynamic servers always say modified today

Content Management Systems

• Most have terrible search engines
• Improve page consistency
  • Standardized presentation
  • Marked navigation (can avoid indexing)
• Force title and other tags
  • Generally in HTML
  • Rarely in PDF and office file properties
• Some can set content date modified tags
• Notify indexer of new content

Levels of Structure in Content

• Structured
• Regular and consistent format
• Defined fields
• Examples: Database records, XML documents
• Semi-structured
• Elements of the document are organized
• Examples: email messages, resumés, some web pages
• Unstructured
  • Ad-hoc human-generated text
  • Examples: many web pages, content of blogs

Multimedia Search

• Non-textual files
• Images
• Audio and music
• Video and Flash
• Numeric data sets
• Use available metadata
• Text on linking pages
• File properties, ID3 content
• Create keywords, abstracts and descriptions
• Extract indexable text
• Closed captions & subtitles
• OCR,speech-to-text

Indexing

• Inverted index
• All words in every document
• Sorted alphabetically
• Document ID, maybe position & context
• Document information
• URL
• Title
• Size, extension and MIME type
• Date (occasionally correct)
• Access permissions
• Other explicit and derived metadata

More Metadata

• Document type identification
• Press releases vs. FAQs
• Entity extraction
• People, places and things
• Context
  • Apple fruit vs Apple Computer vs. Fiona Apple vs. Apple Valley
• Store externally
• Incorporate during indexing

Faceted Metadata Search

• Faceted Metadata
• Metadata about a page or record
• Not just topical
• Date, author, department, content type...
• Enriches the results
• Relevance-ranked list
• Links to interactively filter
• Powerful tool for locating the right stuff

Organized Search Result