SEO TIPS My recent post on how SEO experts started their authority seo blogs proved popular when submitted to Sphinn by Marty at Aimclear (thanks!) and came at a time where frankly I was preparing to give up blogging for a short time to get on with work!

But in my new role as seo nosey-parker (thanks Tim), I thought it would be fun and perhaps useful to examine the seo/sem experts’ robot.txt files, to see if they could offer some insight into how to manage this often neglected file on your site (if you even have one).

What Is A Robots.txt File & Do You Need One?

Rand Fiskin, Jim Boykin, Danny Sullivan (on Daggle), John Andrews, XMCP, Michael Martinez, Marty Weintraub were notable for the lack of a detectable Robots.txt file, so no, it appears it’s certainly not necessary to include a Robots.txt in your website, and I survived without one for long enough.

A Robots file can, simply, tell search engines certain pages of your site not to return in search results crawl. It can also save valuable bandwidth.

What Do SEO/SEM People put in Robots Files?

Hope they don’t mind (and I’ve included the systems they appear to be on in case that’s useful to some) :)

Make A Robots.txt File

For those who don’t know the Robots.txt syntax, there is an * online tool * you can use to make one if you need one.

Out of interest, while investigating this file, I happened across possibly the longest robots.txt in the world? Yours probably doesn’t need to be quite that big (!), and it’s clear a Robots.txt file is indexable by Google and does aquire Google Pagerank (why I nofollowed those links above in case their owners don’t want link love to that file).

And it wouldn’t be fair to talk about seo people’s Robots.txt without mentioning Brett Tabke, founder of my favourite forum, Webmasterworld. Brett uses his as a blog, when actually, his real robots.txt is here.

Update – While some search engines advise you to use to Robots txt to avoide issues like duplicate content, as Sebastian points out in the comments, this is not ideal and perhaps is not the best advice.

Disallow’ing dupes is one of the most common SEO mistakes by the way. Of course search engines encourage this bad practice, but that’s not exactly a good reason to follow the crowd.

Update – Consider these two articles about Robots.txt from some who have spent more time than me thinking about Robots Txt:

Did you know when you link to a Hobo SEO post we have search engine friendly links back to your site if approved? Our comments are also search engine friendly you know (once you've commented on a few posts)! Do you need any more encouragement to get involved in the conversation ;)