The Robots Meta Tag;
<meta name="robots" content="index, nofollow" />
I could use the above meta tag to tell Google to index the page but not to follow any links on the page, if for some reason, I did not want the page to appear in Google search results.
By default, Googlebot will index a page and follow links to it. So there’s no need to tag pages with content values of INDEX or FOLLOW. GOOGLE
There are various instructions you can make use of in your Robots Meta Tag, but remember Google by default WILL index and follow links, so you have NO need to include that as a command – you can leave the robots meta out completely – and probably should if you don’t have a clue.
Googlebot understands any combination of lowercase and uppercase. GOOGLE.
Valid values for Robots Meta Tag ”CONTENT” attribute are: “INDEX“, “NOINDEX“, “FOLLOW“, “NOFOLLOW“. Pretty self explanatory.
- META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”
- META NAME=”ROBOTS” CONTENT=”INDEX, NOFOLLOW”
- META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”
- META NAME=”ROBOTS” CONTENT=”NOARCHIVE”
- META NAME=”GOOGLEBOT” CONTENT=”NOSNIPPET”
Google will understand the following and interprets the following robots meta tag values:
- NOINDEX - prevents the page from being included in the index.
- NOFOLLOW - prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
- NOARCHIVE - prevents a cached copy of this page from being available in the search results.
- NOSNIPPET - prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
- NOODP - blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
- NONE - equivalent to “NOINDEX, NOFOLLOW”.
Robots META Tag Quick Reference
I’ve included the robots meta tag in my tutorial as this is one of only a few meta tags / html head elements I focus on when it comes to Google (and Bing) seo. At a page level – it is quite powerful.
- Title Element – Important – Unique
- Meta Description (optional but advisable in most cases) – Unique
- Robots Meta Tag (optional) – Be Careful
- Canonical Meta Tag (optional – recommended) – Be Vigilant
These tags go in the [HEAD] section of a [HTML] page and represent the only tags for Google I care about. Just about everything else you can put in the [HEAD] of your HTML document is quite unnecessary and maybe even pointless (for Google optimisation, anyway).
If you are interested in using methods like on-page robots instructions and the robots.txt file to control which pages get indexed by Google and how Google treats them, Sebastian knows a lot more than me :)