Track Robot Spiders

• Use your web server log files

• Look for Spiders by "User Agent" field. The biggest ones are:

  • AltaVista	Scooter
    Excite (and WebCrawler)	ArchitextSpider
    FAST (AllTheWeb, Lycos)	FAST-WebCrawler
    Infoseek (Go)	Sidewinder
    Google	Backrub, Googlebot
    Inktomi (HotBot, AOL)	Slurp
    Northern Light	Gulliver

    searchenginewatch.internet.com/webmasters/
    spiderchart.html

  info.webcrawler.com/mak/projects/robots/active.html

• Search the logs for specific pages to see which robots have found them.