Testing Search Indexing Robots : URL Directory Depth

This section is testing how many directory entries in a URL the robot indexing spider will accept. Some systems put a limit on this because there are rogue servers which can snarl the robot in a never-ending loop of links that just get longer and longer. We have twenty levels of depth in this test, which should let us know which spiders can go very deep in the sites, and which draw back.

This page is at Level 4 (number of directories in the URL)

To tell if this page was indexed, search for the code below. This is the "depth" directory, where the directory name is the same for each level. There is also a set of numbered directories: the depth of the page is noted in the directory name and thus the URL path. Searching for these codes should bring up two separate pages, as they are in two separate directory trees.

Search Code: RTest, RTest204, RTestGood, RTestGood204

For more information, see the Directory Depth Master List, and the Search Indexing Test List.

Search Tools Consulting's principal analyst, Avi Rappoport, may be available to help you with selection, indexing and search log analysis, as well as relevance evaluation, user experience testing, and functional search engine work. Please contact us for more information.

Creative Commons - all work copyright © 1998-2009 by Search Tools Consulting.
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 License.