A common question, with a simple answer:

Google can only crawl and index Web sites that the Googlebot can see.

The very next question is: How do I know whether the Googlebot can see my site?

From Google’s Webmaster Guidelines ^[1]*, pay particular attention to the first technical recommendation:

Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.

Lynx is licensed as open source, which means that it is free.  You can either download Lynx v2.8.5 ^[2], or if you don’t want the hassle of downloading and installing it, just use a Lynx viewer ^[3]; you type in your URL and voila – it shows you what the Googlebot can see.

If you are a little more techie and want Lynx on your machine, Tech Republic ^[4]have written a cheat sheet that includes 20 of the most useful Lynx keyboard shortcuts ^[5].

* Site owners: Don’t be put off by the title “Google Webmaster Guidelines” as they are not just for webmasters, they are for Web site owners too, and they are written in simple English.

If you have a site, whether you are in charge of it or contracting it out, you need to make sure it adheres to the basic guidelines; if it doesn’t, your site can be penalized by Google and the other search engines.  And, reading the guidelines is a lot simpler (and smarter) than trying to get your site back into Google once it has been penalized.

Related Stories:

Know what search engines hate and avoid the Google Death Penalty ^[6]

What does the Googlebot see when it visits your site? ^[7]

See your site through different browsers ^[8]