Studies show that web users will only read the top 15 search results for any given query. That means millions of websites gather dust, never to be seenand whats the point of creating that content if youre not read?

Thus evolved the science of search engine optimization, which includes key word counts, editing metatags, link management. Webmasters can spend weeks, even months, fine-tuning these elementsbut like any product, it needs a test drive. Thats what search engine spider simulators are for.

A search engine spider simulator, also known as search engine robot simulator, allow you to see the page as other web crawlers do. Robots is an industry term that describes how Google, et. al scour the Internet for new pages. Theyre like electronic detectives, with each given a particular task. Some bots are designed to follow every link, downloading as many pages as possible for a particular index or query. Others are programmed to look out for new content.

Both these bots play a huge role in whether or not your website ends up in the top 15or languishes at the bottom.

For example, does the bot pick up on your links? Javascript errors can also cause the bot to miss out on important links, and we all know how important inbound links are in search engine ranking.

Does it index every page of your site? Its highly possible that a programming glitch causes the bot to skip a large portion of your content. There goes all your efforts to increase keywords or optimize titles and crossheads!

Its also possible that the bots are basing your ranking on old versions of your website, unable to recognize the changes you have made. You might as well have not done anything at all.

You may have also made the mistake of accidentally blocking a bot from checking a section of your site. While it is important to restrict website users access to sensitive informationfor example, those reserved for a companys internal networks; the personal information of members who have signed up for a newsletter; or premium pages that youd rather reserve for paying subscribersthe bot should be given free rein, if only to improve your chances of getting a higher ranking. If not, thats just like throwing the bot with the bathwater.

It would be impossible to pick up these errors without actually recreating how the bots review your site. You can do this by using a robot simulation software, many of which can be found on the Internet Using the same processes and strategies of different search engines, these programs will read your site and inform you of which pages are skipped, which links are ignored, and which errors it encounters. You can also review the robot.txt files, which will enable you to spot any problems and correct them before you submit them to real search engines.

Youll be surprised at how many things youll find out about web robots, and how the bells and whistles many webmasters include on the site do nothing to improve search engine ranking. For example, search engine robots generally do not see flash based content, content that is made through javascript like javascript menus, as well as content displayed as an image. Youll also be able to monitor how the bots will follow your hyperlinks, very crucial if youre running a big website with sprawling content.

See, thats why robot simulators are a webmasters best friend.