On page optimization: Keep it simple!

by Neha.Grail | 1st October 2010

Search engines employ software programs often called “Bots” or “Spiders” to crawl and find new sites, mostly by following links from the sites that are already known. Thus, search engine spiders crawl and index web pages on the World Wide Web.

If a search engine spider has difficulty in “crawling” the links within your website, then your web pages either will only get partially indexed or will not get indexed at all. This will significantly impact your site’s ranking on the search engine.

Remember: Spiders are unable to follow very convoluted navigation menus.

Why are spiders not able to crawl my website? One of the reasons why spiders may not be able to follow your website is if your website has a complex and convoluted menu.

Search engine bots are prone to “traps” wherein complex URLs/multiple links pointing to each other or the same web page can hinder the spiders from crawling the website effectively.

Example 1: When the spiders are crawling URLs pointing to shopping cart software or content management systems, they are likely to encounter a lot of extra information in the web page URL. Multiple references to session ids or a large number of variables included in the URL can cause crawling issues.

Example 2: Another example is where a website has a large number of URL versions or several URLs redirect to display the exact same web page. In such a case, the spiders are unable to follow the links to multiple versions of the same web page which leads to poor crawling.

What should I do? Simple and short URLs are typically the easiest to crawl. So, you must try to use a content management system that produces short and clean URLs.

Additionally, you must strive to have simplified navigation across web pages, as little flash and graphical objects, and readable text with keywords for spiders to crawl easily. Using sitemaps can also help crawlers to identify and index your web pages.

Related Blogs

Leave a Reply

Name (Required)

Email (Required - will not be published)


Message (Required)