Finding all the right details on the search engine bots can be difficult, especially if you don’t know where to look. There are three major search engines that are highly known all across the world which we’ve talked about constantly, so why not give you the actual details on their spiders, crawlers, bots, whatever you like to call those handy little navigators.

Google

Lets begin with the most popular spider there is; Google Bot. Google Bot is of course associated with Google. This bot is commonly known as Googlebot/2.1, but Google actually has two different bots. There is the one I just listed and also an image spider known as Googlebot-Image/1.0.

When you want to block Googlebot from your site you simply put ‘googlebot’ into your User-agent section of the robots.txt file. Now to actually get crawled by the Googlebot frequently you will have to actually take the time to update your content by at least 1k. If you do this you will discover that your site is being crawled frequently with the chance of even more then once a day depending on how often the page is updated.

This spider/robot can support the navigation of the following types of web media:
HTML SHTML PDF ASP JSP PHP XML CFM
XLS PPT RTF WKS LWP WRI SWF DOC

Currently Google has been said to ignore Meta tags, but occasionally you will see Meta tag’s provided with descriptions of sites in the search results.

You will generally see search results on the following list of sites when you get crawled by the Googlebot:

• Google
• Netscape
• Earthlink
• AOL
• iWon
• Alexa
• Disney
• Go.com

Yahoo!

The next spider that you will want crawling your site happens to be Yahoo!’s. Their bot is known as Mozilla/5.0 with the user agent being ‘Slurp’ when listed in the robots.txt file.

With Yahoo!’s spider it seems to have a bit more difficulty crawling sites with frames making it that you may have a disadvantage when it comes to ranking, but they are slowly but surely getting better.

It can take at least thirty days for your site to be submitted onto the Yahoo! search results after being found by ‘Slurp’ if you do not submit the site on your own. This process could be lengthy, but if you have good quality inbound links then you will be discovered in no time.

MSN

Recently MSN search engine has become known as Windows Live. They tend to be one of the most active spiders out of all the search engines so you will discover they are crawling your site more frequently.

MSN bot has become known as MSNBOT/0.1 with it having the ability to crawl through sites that utilize frames. One way to get listed by MSN is to simply submit your site for free, which could take approximately 30 days after that to be listed.

As you can see with all this information that I just provided you Google seems to make more of their information available compared to Yahoo! and MSN. This truly does make it easier for you to configure your site around what the Googlebot can navigate through getting you listed much faster on one of the largest search engines that is available.