Herding Spiders

Helping Search Engine Robots Help You

Avi Rappoport

Search Tools Consulting / SearchTools.com

Search Engine Strategies

November 9, 2000

• Introduction

• Introducing Robot Spiders

• What Spiders Like

• Flash, JavaScript and Java Links

• Refresh & Redirect Problems

• Virtual Hosts & IP Addresses

• Relative Links

• Site Maps - A Partial Solution

• File Names & File Formats

• Dynamic Pages: ? and & are Red Flags

• Session IDs & Cookies

• Spider-eye View

• Spider Tracking

• Conclusion