Google SEO: Robots Meta Tag
OK – So I’ve theorised about the Title Element, the Meta Description Tag and the pointless Meta Keywords Tag. Next:
The Robots Meta Tag;
<meta name="robots" content="index, nofollow" />
I could use the above meta tag to tell Google to index the page but not to follow any links on the page, if for some reason, I did not want the page to appear in Google search results.
By default, Googlebot will index a page and follow links to it. So there’s no need to tag pages with content values of INDEX or FOLLOW. GOOGLE
There are various instructions you can make use of in your Robots Meta Tag, but remember Google by default WILL index and follow links, so you have NO need to include that as a command – you can leave the robots meta out completely – and probably should if you don’t have a clue.
Googlebot understands any combination of lowercase and uppercase. GOOGLE.
Valid values for Robots Meta Tag ”CONTENT” attribute are: “INDEX“, “NOINDEX“, “FOLLOW“, “NOFOLLOW“. Pretty self explanatory.
Examples:
- META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”
- META NAME=”ROBOTS” CONTENT=”INDEX, NOFOLLOW”
- META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”
- META NAME=”ROBOTS” CONTENT=”NOARCHIVE”
- META NAME=”GOOGLEBOT” CONTENT=”NOSNIPPET”
Google will understand the following and interprets the following robots meta tag values:
- NOINDEX - prevents the page from being included in the index.
- NOFOLLOW - prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
- NOARCHIVE - prevents a cached copy of this page from being available in the search results.
- NOSNIPPET - prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
- NOODP - blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
- NONE - equivalent to “NOINDEX, NOFOLLOW”.
Robots META Tag Quick Reference
| Terms | Googlebot | Slurp | MSNBot | Teoma |
|---|---|---|---|---|
| NoIndex | YES | YES | YES | YES |
| NoFollow | YES | YES | YES | YES |
| NoArchive | YES | YES | YES | YES |
| NoSnippet | YES | NO | NO | NO |
| NoODP | YES | YES | YES | NO |
| NoYDIR | NO | YES | NO | NO |
| NoImageIndex | YES | NO | NO | NO |
| NoTranslate | YES | NO | NO | NO |
| Unavailable_After | YES | NO | NO | NO |
I’ve included the robots meta tag in my tutorial as this is one of only a few meta tags / html head elements I focus on when it comes to Google (and Bing) seo. At a page level – it is quite powerful.
- Title Element – Important – Unique
- Meta Description (optional but advisable in most cases) – Unique
- Robots Meta Tag (optional) – Be Careful
- Canonical Meta Tag (optional – recommended) – Be Vigilant
These tags go in the [HEAD] section of a [HTML] page and represent the only tags for Google I care about. Just about everything else you can put in the [HEAD] of your HTML document is quite unnecessary and maybe even pointless (for Google optimisation, anyway).
If you are interested in using methods like on-page robots instructions and the robots.txt file to control which pages get indexed by Google and how Google treats them, Sebastian knows a lot more than me 
External Links
- http://googlewebmastercentral.blogspot.co.uk/2007/03/using-robots-meta-tag.html
- http://www.robotstxt.org/meta.html
- http://googleblog.blogspot.co.uk/2007/02/robots-exclusion-protocol.html
Written by Shaun Anderson
Hi there, I thought that all these meta keywords – description are useless??
No the Meta Keywords are next to useless for me – but the meta description is very important to me.
hi shaun
Im not sure if ive got meta tags on my site as i was advised they are old hat – is this true ?
If tags added to my site (ive no idea how to do this ) would it affect its current performance ?
Martin
Howdy Martin you’re learning how to get a hold of me
You’ve got nothing to worry about. Though I would modify your page titles a bit looking at them. Don’t repeat the keywords as much.
PM Alan for help.
You don’t need to worry.
Now, we need to add one more tag to avoid duplicate content, which is recently announce by three giants, google, microsoft and yahoo.