why I have one.. i simple I wanted fasted indexing from ASK that’s the only reason

I did Build a cool tool for making robots.txt files.. ages ago

http://www.clickability.co.uk/robotstxt.html

Read Sebastian to Shaun’s – The Idiots Guide To Robots.txt

To follow in Sebastian’s footsteps here is something on robots.txt and pagerank accumulation

The fun thing is when people have a “secret” area on their site that they don’t want indexed, so they block it in the robots.txt file rather than using a more secure method.

Yeah tell me about it! Lets just say Sebastian has educated me via email to the wonders of robots.txt since this post, and has sent a couple of hundred stumblers and twitters to witness my slapping

Congrats on top spot on the best SEM forum on the net by the way and for keeping it – marvelous spin!

Actually, I am getting as sick of seeing your mug up there as I did Rand Fiskin’s although I am on the greatest hits myself (through sheer good fortune with Aaron Wall’s article).

Also seriously, old-timers know to autogenerate the robots.txt so what you see when you pull it is not (necessarily) the same robots.txt the search engines get. We used to actually show incorrect ones to everybody but qualified search spiders — to consume competitors valuable time — but I don’t know how many people still bother. From your list of popular bloggers, I doubt any do serious competitive work on their public blogs anyway.

The way I figure it, you added more *actual* useful information to this page (which was just a bit of link-baiting fun crafted in the middle of the night) – so you deserve credit in the only way I can give you – a link.

But saying that, that’s enough spamming of my site Sebastian.

Away with you!

And plz add nofollow crap to Kalena’s and Andy’s robots.txt links.

Hand slapped, Hang Head In Shame

I’m sorry, that’s plain false. Disallow’ing files in robots.txt doesn’t prevent from indexing, the Disallow: statement just forbids crawling. Hence it’s a bad idea to steer indexing in robots.txt.

The current robots.txt commands (Disallow:, Allow: and Sitemaps:) are all crawler directives that control crawling but not indexing. Besides Google’s Noindex: experiments (somewhat flawed I may add) there’s no such thing as an indexer directive in robots.txt. Search engines do index URLs that are disallow’ed, and that’s the right thing to do. Disallow’ed URLs do accumulate PageRank, and appear on SERPs as URL-only listings or with titles and snippets pulled from 3rd party sources.

Disallow’ing dupes is one of the most common SEO mistakes by the way. Of course search engines encourage this bad practice, but that’s not exactly a good reason to follow the crowd.

However, “outing” robots.txt files is a great idea, and thanks for the mention.

Best

Lee Giles