Tuning Search: Analytics, Search Logs and Best Bets


Avi Rappoport

Search Tools Consulting




Tuning Search



Metrics - Indexing

Metrics - Search Traffic

Analytics - Most Popular Queries

zipf curve






Example Query Report - List

list-style search report

This report from WebSideStory, formerly Atomz

Example Query Report - Billboard Chart

billboard-style chart Tracks the number of queries and changes from last week, also the number of weeks that query has been on the list.

This report generated in Excel by Search Tools Consulting









Example Query Report - Graph

graph of top queries

ipod, laptop, mp3, ipod, mini, mp3 player, digital camera, notebook, mobile phones, dvd player, dvd recorder, navman, printer, acer

This report from SLI Systems











Example Query Report - Future May Be Interactive

namevoyager example

NameVoyager at BabyNameWizard.com

Java app that allows interactive exploration of names over time

Good way to handle huge amounts of text

Report from NameVoyager






Popular Query Terms - Search Analytics


Analytics - Navigational Search


example: FedEx forms

Search Logs - Query Format Requirements

If storing IP addresses, define a privacy policy to avoid inadvertent release of personal information.


Search Logs - Click Tracking Format

Example Log File, courtesy of Librarian's Index to the Internet

  • 2006-04-01:00:09:18 sex education 33 http://lii.org/ search=sex education action=show searchtype=keywords
  • 2006-04-01:00:09:32 chemistry 91 http://www.lii.org/pub/htdocs/ search?action=show;search=chemistry;searchtype=keywords page=2 action=show search=chemistry
  • 2006-04-01:00:10:50 "educational technology" 14 http://lii.org/pub/htdocs/search ?search=sex%20education;action=show;searchtype=keywords action=show search="educational technology" searchtype=keywords
  • 2006-04-01:00:10:56 <A HREF=http://0-day.biz/ebay/index.php>Coupons for eBay</a> 0 search=<A HREF=http://0-day.biz/ebay/index.php>Coupons for eBay</a> action=show searchtype=keywords
  • 2006-04-01:00:11:19 laxative 4 http://lii.org/pub/htdocs/search action=show search=laxative searchtype=keywords
  • 2006-04-01:00:12:10 laxative contain bark 0 http://lii.org/pub/htdocs/ search?action=show;search=laxative;searchtype=keywords action=show search=laxative contain bark searchtype=keywords
  • 2006-04-01:00:12:33 chemistry 91 action=show search=chemistry searchtype=keywords
  • 2006-04-01:00:12:33 biography of ernesto che guevara 0 http://www.lii.org/pub/topic/history action=show search=biography of ernesto che guevara searchtype=keywords
  • 2006-04-01:00:12:48 laxative 4 http://lii.org/pub/htdocs/search?action=show; search=laxative%20contain%20bark;searchtype=keywords action=show search=laxative searchtype=keywords

Analytics - No Matches Found


Leveraging No-Match Analytics


Example of Special-Case Results

search results

Bloomingdales found many queries for Nine West shoes, so added a note and results for similar shoes

Results Click-Through Analysis


Results Next-Page Analysis


Analyzing the Long Tail

  • 20% or more query terms are unique (depending on the period analyzed)
  • Many of the rest are very rare
  • Some have no matches - analyze those separately
  • Look for broader categories
    • Numbers and product IDs
    • Domain names
  • Break out individual words
    • Look for words not in most popular term queries
  • banning of slaughterhouse five
  • Banning The Catcher in the Rye
  • baseball girls
  • basegirls
  • battery
  • bellaonline
  • Bernoulli
  • Betty Friedan
  • bi-lingual education in schools
  • blogs
  • Bob Fosse

Session Analysis

Conclusion - Tuning Search