Tuning Search: Analytics, Search Logs and Best Bets

Avi Rappoport

Search Tools Consulting

Background

• Avi Rappoport
  • Librarian and long-time search engine analyst
  • Fascinated by user behavior
• Search Tools Consulting
  • Editor & Publisher, SearchTools.com
  • Research, analysis, dissemination
  • Real-world consulting
• Search Log Analysis
  • Like peering into users' minds... through a kaleidoscope
  • Fascinating surprises
  • Strong similarities over time and across sites

Tuning Search

• Search is not a turnkey tool
  • Each enterprise will have special needs
  • You won't be able to guess them
• Tools to help you
  • Metrics
  • Search logs to generate reports from
  • Popular Query Reports
  • No-Matches Query Analysis
  • Results Click-Through Analysis
  • Results Next-Page Analysis
  • Analyzing the Long Tail
  • Session Analysis
• Rewarding investment in the long run

Definitions

• Analytics
• Pay attention to the ongoing processes
• Simple metrics
• Complex analysis
• Search Logs
  • Raw material for text mining
  • Not the same as web logs
  • Include search-specific information
• Search Suggestions
• aka "Best Bets"
• Supplement algorithmic logic with human judgment
• Display to users when they search for specified terms

Metrics - Indexing

• Number of pages or records indexed
  • Duplicates
  • Errors - keep an eye on this!
• Increase in pages
  • New site or section
  • Improvements in web publishing
    • better linking
    • break topics into separate pages
• Decrease in pages
  • Hardware or software problem on a server
  • Adding access control (passwords)
  • Publishing changes
    • Java or JavaScript links

Metrics - Search Traffic

• Average Usage
• Per day
• Per week (weekends are often lower)
• Per month
• Traffic Changes
• Increase
  • Generally good
  • Sometimes weird bot or DOS attacks
• Decrease
  • Search engine broken?
  • Site usage down?
• Not Just Queries
  • Going to second and subsequent pages
  • Clicking on result link
  • Help clicks

Analytics - Most Popular Queries

• Popular queries are an amazing proportion of query terms
  • Top 1% of query terms can be 20% of total
• We find this distribution in most search engines
• Pareto's 80/20 rule
• Zipf distribution
• "The Long Tail"

Example Query Report - Billboard Chart

Tracks the number of queries and changes from last week, also the number of weeks that query has been on the list.

Example Query Report - Graph

Example Query Report - Future May Be Interactive

NameVoyager at BabyNameWizard.com

Java app that allows interactive exploration of names over time

Good way to handle huge amounts of text

Popular Query Terms - Search Analytics

• Top queries sometimes suprising
  • Short queries more popular by default
• Empty queries
  • Users just click "Search"
  • May want to bring up error dialog
• Navigational Search
• Use these for Search Suggestions
  • Concentrate on most popular terms
  • Suggestions should clearly describe target pages
  • Avoid marketing jargon
  • Track changes
    • Query terms
    • Site coverage

Analytics - Navigational Search

• Approximately 1/3 of users are search-dominant
• Even when a link is on the front page in big red letters
• Make sure search field is on every page
• Relax and live with it
• Search avoids site architecture problems
• Long lists
• Deep interactive drill-downs
• Confusing terms (especially for non-native speakers)
• Work with IA on improving user interaction

Search Logs - Query Format Requirements

• Standard and consistent - tab delimited is best
• Timestamp
  • Date & time, offset from GMT (for consistency)
• Query terms
  • Original query terms- as entered by the user
  • Normalized query terms - lowercased, stemmed, field names removed
• Other Parameters
• Character set, Language, Field names, Date ranges, Zone...
• Page or starting result number
• Referring page, URL "referer"
• Cookie, Session ID or IP Address

Search Logs - Click Tracking Format

• Click tracking requires special URLs in results pages
  • Search engines should provide automatically
  • Send results click to search engine for logging before loading page
• Search log entry
  • Include query log info
  • Clicked URL
  • Whether it's a search suggestion or algorithmic result
  • Result number

Analytics - No Matches Found

• Percentage of no-matches varies
  • Search default: match any word, all words, phrase
  • User expectations of site
• Scope problems
  • Typing domain names
  • Extremely general queries
• Vocabulary Problems
• Spelling and Typing Errors
• inmigration
• valentines day
• pokemon vs. pokémon
• grandcanyonhotel

Leveraging No-Match Analytics

• Remove search case-sensitivity
  • Treat lowercase, Mixed Case and UPPERCASE the same
• Change default match algorithm
  • Commerce sites may want to show a match for any term
• Add a spellchecker
• Add vocabulary
  • Reduce use of jargon, substitute user vocabulary
• Add synonyms in metadata
• Improve the no-matches page
• Expand the searchable content
  • Index more content, add new topics or products

Example of Special-Case Results

Results Click-Through Analysis

• Query Click-through Ratio
  • 1:1 is ideal
  • Low ratio - the results are unappealing
  • High ratio - the results are not useful
  • Use session analysis to learn user intent for low or high ratios
• Top Pages Clicked On
  • Generally first position in results or search suggestion
  • Have clear title, search terms in context
• Fixing problems
  • Improve titles
  • Add Search Suggestions
  • Add user vocabulary terms to text or keywords
  • Adjust search results weighting algorithm

Results Next-Page Analysis

• Query - Next Page Click Ratio
  • Lower is always better
  • Higher means the results don't look useful
• May be a scope problem
  • Site simply does not include certain topics
• Concentrate on core topics
  • May be multiple meanings for a term or acronym
  • Term is too generic, doesn't match enterprise name
  • Add Search Suggestions
    • Highlight various options
    • Suggest landing pages

Analyzing the Long Tail

Session Analysis

• Watch user interaction with search engine
  • Learn more about their real information need
• Integrate with web site clickstream and purchase or other metrics
• Example:

timestamp	query	hits
2006-03-30 12:07:10	volcanoes in the philippines	1
2006-03-30 12:07:58	volcao	0
2006-03-30 12:11:41	volcano's	27
2006-03-30 12:12:15	volcano costa rica	0
2006-03-30 12:12:18	geologic history	36
2006-03-30 12:12:35	volcanos in costa rica	0

Conclusion - Tuning Search

• Quality search requires attention and maintenance
• Metrics are good but not enough
• Investing in search log analysis pays off
  • Top query reports
  • No-matches reports
  • Click-throughs and next pages
  • Thinking about the long tail
  • Session analysis for user intent
• This presentation online at SearchTools.com/slides/ess06/tuning-search.html