My recent post on how SEO experts started their authority seo blogs proved popular when submitted to Sphinn by Marty at Aimclear (thanks!) and came at a time where frankly I was preparing to give up blogging for a short time to get on with work!

But in my new role as seo nosey-parker (thanks Tim), I thought it would be fun and perhaps useful to examine the seo/sem experts’ robot.txt files, to see if they could offer some insight into how to manage this often neglected file on your site (if you even have one).

What Is A Robots.txt File & Do You Need One?

Rand Fiskin, Jim Boykin, Danny Sullivan (on Daggle), John Andrews, XMCP, Michael Martinez, Marty Weintraub were notable for the lack of a detectable Robots.txt file, so no, it appears it’s certainly not necessary to include a Robots.txt in your website, and I survived without one for long enough, but it can help reduce duplicate content issues if you’re worried about that sort of thing and not confident enough to hack away at your template to physically prevent problems.

A Robots file can, simply, tell search engines certain pages of your site not to return in search results crawl. It can also save valuable bandwidth.

What Do SEO/SEM People put in Robots Files?

Hope they don’t mind (and I’ve included the systems they appear to be on in case that’s useful to some) :)

Make A Robots.txt File

For those who don’t know the Robots.txt syntax, there is an * online tool * you can use to make one if you need one.

Out of interest, while investigating this file, I happened across possibly the longest robots.txt in the world? Yours probably doesn’t need to be quite that big (!), and it’s clear a Robots.txt file is indexable by Google and does aquire Google Pagerank (why I nofollowed those links above in case their owners don’t want link love to that file).

And it wouldn’t be fair to talk about seo people’s Robots.txt without mentioning Brett Tabke, founder of my favourite forum, Webmasterworld. Brett uses his as a blog, when actually, his real robots.txt is here.

Thanks for visiting the Hobo Blog! Why not subscribe to our Feed for free internet marketing tips! We really welcome any feedback on our articles, so don't be scared to leave a comment either!