Mon 11 Feb 2008
Testing Matt Cutts’ & Google ‘Minty Freshness’
Posted by Shaun AndersonSo after my recent posts weren’t appearing immediately in Google’s main web search, I thought I’d post a quick article just to test if my latest article would indeed appear in minutes as was the case up until the beginning of February 2008. Matt Cutts confirmed some time ago this is how Google was working these days.
It was a clear phenomenon, quite distinct from how Google has been treating this blog recently. Rankings not changed, traffic the same - it’s only the way Google is treating the blog ‘area’ that is/was different from 2 weeks ago.
I go into it in more depth in precious articles;
- Has Google Canned The Fresh Content Boost ‘Experiment’?
- Meet The Google Fresh Content Blog ‘Penalty’!
A few of my online friends tested on their own blog and only Jeff at search engine people seemed to be witnessing a similar phenomenon with a recent post.
Barry Schwartz at SEL commented;
“In my opinion, it is just the nature of blogs not being updated for a while and then Google stops crawling it as often”
….hmmm he might not have actually read the post here (as he linked to the Sphinn discussion and not this blog, the scoundrel!) but as he can see from looking at the dates I post on this blog has been very active the last month or so, so that is patently not the cause. I think I have posted 37 articles since January 1 2008.
Tim Nash recently reported all was well, so it certainly was not a widespread phenomenon - it seemed localised here with the only visible similarity being Jeff.
Ian offered some advice;
“I’m aware of a site that Google indexes every day and content appears in the SERPs within hours of posting… but for years Google takes a holiday from indexing the site for a couple of days once every month or two… see what happens over the next few days.”
…sounds intelligent advice, which this seo has taken.
This post will allow me to test the ‘minty freshness’ of Google, at least when it comes to this blog, and help me determine whether something is amiss either with Google or closer to home. It’s either go to be something amiss with my WP install or something do to with a change in how Google indexes this blog.
So what I have is a blog which up until recently saw posts immediately feature in Google. Now it takes hours usually, at the height of this phenomenon it took nearly 24 hours.
Here goes (I’ll edit this post with my analysis)…
- Google Web Index
- Google Blog Search
- 9 Hours?
- Feedburner
- Minutes - Technorati
- Minutes
This post was posted at [Mon, 11 Feb 2008 19:06:26]
- Update; 1/2 Hour Later Still no Google minty freshness and surprisingly not even in Google blogsearch
- Update; 1 Hour Later still not in Web Search or Blog search
- Update; 2 +1/2 Hours later Google still blissfully unaware of the page (apparently)
- Update; 3+1/2 Hours later still not in Google
- Update; 4 +1/2 Hours later still nowhere
- Sod’s law had to restart Apache - Outage for 1 hour damn it!
- Update; 5 +1/2 Hours later still not in Google
- Update; 6 +1/2 Hours later still no Google recognition
- so nearly 9 hours later not in Google
- Update; 20 Hours later still no appearance in Google’s web search
- Update; 2 days later…..nowhere
mmmm…. not exactly what I would call minty fresh indexing.
I called this a “fresh content penalty“. Perhaps it is simply a phenomenon, but when a change this great happens only affecting new content, with no loss of Google traffic or SERP rankings, it’s curious to say the least. I’ve checked all is well site-side and it seems it is.
It’s not a ‘penalty’ because it hasn’t happened to you yet, believe me, but it’s a major change in the way Google is working this blog. If this were a low authority seo news site, for instance, this blog would be scuppered in the serps at least. I see more authoritative sites like SEL and Sphinn constantly injecting new material into the serps, so I know it is not widespread, and would question if those sites would actually fail an “authority score” anyway.
Perhaps Google has taken a holiday from this site as Ian said and this is just a phenomenon I haven’t witnessed yet -perhaps something is wrong this server side - either way it’s worth examining for the next blog it happens to, especially as things are far from resolved.
Perhaps this site doesn’t meet some new “authority” status although when Google does rank the post a day later it’s usually top ten.
Thanks for visiting the Hobo Blog! Why not subscribe to our Feed for free internet marketing tips! We really welcome any feedback on our articles, so don't be scared to leave a comment either!

That is strange, from tests done in-house we have seen something very different to that what you are seeing!
Try a different test, this time make a new blog, on http://www.blogspot.com and do some ping’s and gauge the success you have there. Then make a Digg post, and gauge the success there, i’m 90% sure what you are seeing is an anomaly and you should definitely see what you expect to see.
As a quick question, you are pinging this post to the common places right? And you do have a sitemap, with which you pinged the major search engines to tell them that you have made an update? You can’t just expect them to check your website all the time, it is very dependant on when the last crawl was! Crawl cycle average is 0.5 days between crawls even for the largest/authoritive of websites like engadget.com (source: seometer.com). If you ping your sitemaps i would assume a MUCH quicker discovery of your URL in the serps.
Comment by William - Neutralize — February 12, 2008 @ 12:25 pm
Yup, tested it today, Google’s BlitzIndexing is defunct or hates my blog all of a sudden. Google’s Web search managed to index my Sphinn user profile (the RSS update) faster than my breaking news.
Comment by Sebastian — February 12, 2008 @ 11:20 pm
Ok, the requested test is done. Time to index = 1.5 hours. That’s lame. It used to be a few minutes.
Comment by Sebastian — February 12, 2008 @ 11:51 pm
I’m glad others are noticing it’s not as fast as it was. This post still isn’t in nearly 2 days later….
Comment by Shaun Anderson — February 13, 2008 @ 1:07 am
@ William thanks for the comment but the sites you mention have an authority all of their own and would not be subject to the sort of “lack of authority” score which I would think would be the obvious reason for a change in how Google handles smaller blogs.
We’re not likely to see search engine land or SERoundtable suffer either. I’m taking about the smaller seo blogs of which there are plenty.
I don’t have a xml sitemap on this site - never needed one.
@ Sebastian sorry to hear keep an eye on it over the next few days
Comment by Shaun Anderson — February 13, 2008 @ 1:13 am
As mentioned by email, are you using pingshot, or your own ping list?
In my blogsearch exploration yesterday I saw a huge difference between what Google Blogsearch and Technorati were listing, though it didn’t seem quality related.
Comment by Andy Beard — February 13, 2008 @ 3:12 am
Hi Andy sorry I meant to get back to you. I think I am just pinging pingomatic but now
I’ve turned on Pingshot so I will see if that makes a difference - thanks.
It doesn’t change the fact something’s changed as I don’t really tamper about ‘back there’ so to speak.
Comment by Shaun Anderson — February 13, 2008 @ 3:20 am
I just don’t think you can expect Google to come here that often, they do go through 20 petabytes indexing per day but unless they think you have updated then they won’t come back, i think once every couple of days for a blog post on a smallish blog is reasonable! And they do allow you to ping them with your updated pages if you want!
I would recommend that your next step is to add a sitemaps, and see if that helps
Comment by William - Neutralize — February 13, 2008 @ 10:23 am
Hi William I only expected Google because that was the norm for 6 months previous.
Why would I need to add a xml sitemap? Up until a couple of weeks ago everything was rosy…. I shouldn’t need to change anything if nothing’s changed in the back end of this blog or at the Big G’s end….
Comment by Shaun Anderson — February 13, 2008 @ 2:54 pm
I mean, to see if this is actually a problem or whether Google has made a tweak to their algorithm which determines what time a site should be rechecked. It may be a cost saving exercise, which would make sense and then if you tell them a page is definitely new (via a sitemap) they would be more like to go and index it. If you think about it, it does make sense that an authoritative website gets indexed quicker….
Obviously this is just a random conclusion I have came to, however just because Google isn’t indexing your website does not mean its a G problem, its probably more than likely a algorithm change and you shouldn’t focus on why it has happened, but more so how to get around the problem
Comment by William - Neutralize — February 13, 2008 @ 3:21 pm
I consider myself a smaller blog, but I’m not seeing this penalty yet. Maybe because mine is still relatively new.
Comment by SlightlyShadySEO — February 14, 2008 @ 3:03 am
Hi Shady
Of course, it might not be a penalty at all but it sure feels like it. I’m double checking all I can at this end - it’s quite time consuming. I’m updating my version of Wordpress and my plugins first.
Comment by Shaun Anderson — February 14, 2008 @ 3:36 am
Noticed a similar issue with Fused Nation lately as well.
New posts are included the regular index almost instantly (no cache though) but not blog search (they were previously). It’s at the point now where I have around 5 posts on different blogs referring to my post that appear in blog search (for “post title”), but my original post doesn’t appear.
A blog search for “fusednation” returns no results, although “fused nation” does - but no pages from the site just the URL as a related blog. It even shows a single reference on a comment on DaveNs blog but no pages from my blog.
I’m not certain that authority / PageRank is related to the issue though - there are blogs with lower (TB) PR and have been around less time show up refering to my posts.
Offhand it seems that the site is simply removed from Blog Search results (but still returns as a result for a site name search).
No obvious reasons why - the site has a reasonable amount of links - new posts regular get linked to from other blogs - content is unique, etc.
No issues in Google sitemaps - a reasonable amount of Google feed subscribers, no content issues, variety of IBL text, sitemap downloaded regularly (and recently rebuilt it due to new design). Plus the site has had over 15,000 visitors already this month (but only a small percentage from organic referrals).
The only thought I’ve had is that the sitemap generator plugin produces an error trying to ping Google (when new posts are posted or old ones are edited). I don’t see how this would result in the blog being removed from blog search though - perhaps only affect new pages being included. Given that new content can be live in regular search within minutes, I’m assuming the ping works though.
Seems to be a new thing - only really noticed it since I changed the WP version to include G Blog incoming links instead of technorati. Can’t remember when that change was made, but within the past few months I think?
Scott
Comment by Marketing Guy — February 14, 2008 @ 3:44 pm
These things can drive us bloggers mad with worry. There’s nothing like breaking a story only to see it sit there unnoticed by search for days. I’ve personally had good luck with fast indexing, but I have many friends that tear their hair out with frustration while their hot news cools during a long wait.
Comment by Jeff Rivera — February 14, 2008 @ 5:03 pm
I have the feeling that Google is slowing down on a lot of things.
Here is an example of Google being VERY slow in updating itself:
A site (home page) disappeared on the SERP’s 12 Feb due to other sites copying it. We changed the content and Google recached 15th Feb. 19 Feb the snippet showed the update, but the cache still showed the old version. Now 20 Feb, and the Google cache correctly shows the updated version, but when I search for unique text on the page, the site still does not show. The top SERP’s have not yet come back.
A month ago when the same thing happened, the site was recached the next day, and back in the top SERP’s later that day.
VERY slow at the moment.
And as an aside - what is your experience with scraper sites copying content and destroying results. Do you have a post on this - so this post does not get totally off topic
Comment by Search Engine Optimisation-Michael — February 19, 2008 @ 9:58 pm
You know I noticed a change, but only after my hosting company insisted I add the WP-Cache Plugin to my blog. Anyone else notice this?
Comment by Melissa - SEOAware.com — February 21, 2008 @ 6:04 am
Hi Shaun
I have seen this same issue with one of our sites - very recently.
Quick question - in Google Webmaster Tools (GWT), does the main page for your blog - http://www.hobo-web.co.uk/seo-blog/ - appear in list of pages with INTERNAL links?
We have a site that exhibiting the behaviour you mention - blog results not showing up instantly in Google but its still getting crawled every couple of days etc. But interestingly, the root for the blog is not in the list of internal pages - despite there being many links to this page. Other individual blog posts that are cross referenced/deep linked are in the list - just not the root page.
Matt
Comment by Matt Hopkins - Vertical Leap — February 21, 2008 @ 8:02 am
Thanks for the comments guys.
@ Matt - No according to GWMT the page is there with all my internal links showing up so that’s not the issue I’m afraid. Thanks for the comment - hope you stick around (hint) my next post is a cracker!
Comment by Shaun Anderson — February 21, 2008 @ 9:45 am