In addition to server-wide robot control using robots.txt, web page creators can also specify that certain pages should not be indexed by search engine robots, or that the links on the page should not be followed by robots. The Robots META tag, placed in the HTML <HEAD> section of a page, can specify either or both of these actions. Many, but not all, search engine robots will recognize this tag and follow the rules for each page.
Task Entry Notes Do not index, but follow links
<META name="ROBOTS" content="NOINDEX">Use this for pages with many links on them, but not much useful data. Because "follow" is the default, you don't have to include it. Index, but do not follow links
<META name="ROBOTS" content="NOFOLLOW">Use this for pages which have useful content but links which may be irrelevant or obsolete. Do not index or follow links
<META name="ROBOTS" content="NOINDEX,NOFOLLOW">This is for pages which should not be indexed at all. If you put that in every page, the site should not be indexed. Index and follow links
<META name="ROBOTS" content="INDEX,FOLLOW">This is the default behavior: you don't have to include this tag. Note: if you add Robots META tags to a framed site, be sure to include them on both the FRAMESET and the FRAME pages.
For more information, see the original HTML Author's Guide to the Robots META tag and the SearchEngineWorld Robots Meta Tag page.
- SearchTools Robots Testing - test cases index page.
|
|
|
|
|
|
|