How do I make sure the search engines can crawl my site?

The relevancy of a website is one of the few controllable forces a business can impact in its search to climb the elusive rankings, increase visibility and chip away at share of traffic. Insuring a site is as search friendly and content rich as possible is one of the most fundamental factors at serving the search engine as well as website visitor.

There are several best practices which insure a website is technically as well as visitor search friendly, here are just a few*:

Factors:

Website Performance Auditing:

*GDMI offers Website Performance Audit services to specifically evaluate the technical factors which may be limiting a site’s search ‘ability’.  Free webmaster tools are also available such as Google’s which can provide additional insights to making a site attractive to the search engines.

Publishers and portals sites whose main purpose is to attract readers and visitors to the content it provides on an ongoing basis, by definition would be expected to be among the highest ranking websites online.  After all content is king and a content driven website should have the best opportunity for achieving such rankings. So why aren’t more newspapers and informational websites and portals more visible more often?

The architecture of many websites can actually preclude a site from being fully crawled and ‘read’ by search engines. Content is often archived behind non search friendly content management systems that search engines cannot access therefore will not account for in determining relevancy.

Results:

At the end of 2006 GDMI began working with a tourism, travel and entertainment publisher who had recently migrated its offline magazine to an online portal to rival such properties as Metromix.com. The publisher commissioned numerous writers, editors and photographers from the offline magazine to contribute to the ongoing flow of editorial content for the online version of the magazine.

The manner in which the site stored the articles and the content management system used did not allow for previous issues or articles to be crawled or indexed. 

At the onset of the campaign, GDMI was retained to increase traffic and visibility to the site utilizing paid search traffic. As a key factor in gaining advertiser support, the site needed to have as much visibility as possible.

Several interim solutions were considered to build out certain areas of the site with individual search friendly HTML pages or to undergo a considerable off site visibility campaign however it was ultimately determined that these would not be scalable solutions over the long term. Getting the ‘house in order’ was determined as the most cost effective strategy for building a solid online foundation.

Therefore in 2007 GDMI redesigned, reengineered and redeployed the website building it on search friendly technology, Ruby on Rails, which would not only enable the articles and abundant content to be crawled and indexed by the search engines but also allowed for a user friendly interface to be sustained by the client’s internal staff.

Other search engine friendly enhancements made to the site included:

The incremental value of the undertaking was an overall increase in collective pages indexed among the top three search engines from 5,982 in January 2007 to 33,725, July of 2008.  A more than 460% increase in the overall volume of pages which became visible, searched and indexed by the search engines.

Not every site may require such an overhaul, but every site can be made more relevant and more search friendly for the ultimate goal of increased visibility and traffic from search engines and users alike.