No “size matters” jokes here, just straight to the info. On Monday, Tim Mayer announced in Yahoo’s blog that they have increased their index – and not just a little, by a lot. Yahoo! now claims to have 19 billion documents in their index, which is up from a “meager” 6 billion that had last week. Compare this to Google’s 8 billion. Now, before you get all impressed with the work that Yahoo! has put in to make them the “most comprehensive search engine” available, know that one of the worst and stupidest bots out there is Slurp (Inktomi).

A quick history lesson: Back in 2000, as Yahoo! was having their search results (along with HotBot and others) served by Inktomi, Yahoo! saw a great opportunity and put in an offer to by Inktomi for $1.3 billion. The execs at Inktomi wanted more despite the recent “dot bomb” crashes of the spring of 2000. So, what did Yahoo! do? They dumped Ink and went with Google to serve their results. Inktomi was unfazed about it in the media, but behind the scenes they were scrambling. They went through numerous layoffs, cutbacks and such (i.e. Slurp was NOT updated, upgraded, etc.), and then two and a half years later, Yahoo! puts in a bid, much lower than their original, to buy Inktomi whcih is accepted. Through this time, Inktomi was not touched, it was the same bot, despite the upgrades in technology and code.

To continue, Yahoo! sits on Inktomi for about a year, does nothing with their new bot they have bought and then launches Yahoo! Search. Have they upgraded Inktomi since then? No. So, what I have done recently is build sites with very basic and routine CMS (Content Management Systems) that are not complex, but did some redirects and served dynamic content. The results? MSNBot and GoogleBot spidered the pages without a hitch. Slurp? It was caught in many spider “traps” which didn’t serve session IDs and were very basic. In looking at MarketLeap, Yahoo! would have 2-4 times the number of pages indexed. Now, even though there were no session IDs, it was clear that Slurp was confused by the dynamic content and it would index the same page multiple times thinking it was a different page.

It is clear that the new huge index of Yahoo! is filled with at least 30% of duplicate content pages.

Yahoo! Hand-Editing SERPs
We were alerted to a good thread in the Webmaster World forums regarding if Yahoo! was indeed, hand-editing the SERPs. In our testing, we do screen captures of the SERPs, so we can verify, not just the SERPs, but also the paid listings, the layout, structure, etc. It is what sets us apart, I believe, we have an extensive library of screenshots to refer to. Now, when I read this thread, I couldn’t see the value in Yahoo! spending the time in hand-editing certain keyword phrases, but in looking through the logs, this is exactly what is happening. Google is kicking the teeth of Yahoo!, repeatedly. With Slurp not being updated, and their index bloated because of it, Yahoo! is spending money left and right as if it is trying to solve its problems by throwing money at it, which we all know doesn’t work.

If Microsoft can get its act together, which they seem to be doing, they should pass Yahoo! in market share by year’s end and it should be just a two engine race for dominance between MSN and Google. Unless Yahoo! makes changes, cleans up Slurp and actually have an algorithm that is fine without hand-edits, they are headed in the same direction as HotBot and Infoseek. Once a great and widely used search engine, now in the junk heap.

I can’t believe I am going to say this: Based on our testing results, it is our recommendation that you cease in optimizing for Yahoo! and concentrate soley on Google and MSN. You read that correctly. Too many issues with the logs and the testing results to allow for any more time spent on Yahoo! as a primary provider of qualified traffic. Yahoo! is not in good shape. Concentrate instead on using PPC in Yahoo! instead of the organic listings. If a Yahoo! directory listing comes up for renewal, let it expire, unless you are getting 3,000 or more referals from it per year. It just isn’t worth it anymore. Yahoo! Search may be dying.