A common question, with a simple answer:

Google can only crawl and index Web sites that the Googlebot can see.

The very next question is: How do I know whether the Googlebot can see my site?

From Google’s Webmaster Guidelines*, pay particular attention to the first technical recommendation:

Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.

Lynx is licensed as open source, which means that it is free.  You can either download Lynx v2.8.5, or if you don’t want the hassle of downloading and installing it, just use a Lynx viewer; you type in your URL and voila – it shows you what the Googlebot can see.

If you are a little more techie and want Lynx on your machine, Tech Republic have written a cheat sheet that includes 20 of the most useful Lynx keyboard shortcuts.

* Site owners: Don’t be put off by the title “Google Webmaster Guidelines” as they are not just for webmasters, they are for Web site owners too, and they are written in simple English.

If you have a site, whether you are in charge of it or contracting it out, you need to make sure it adheres to the basic guidelines; if it doesn’t, your site can be penalized by Google and the other search engines.  And, reading the guidelines is a lot simpler (and smarter) than trying to get your site back into Google once it has been penalized.

Related Stories:

Know what search engines hate and avoid the Google Death Penalty

What does the Googlebot see when it visits your site?

See your site through different browsers
 

Sphere: Related Content

19 comments:

  1. matinee
    24 August 2007 0:05

    Hi all,

    Can any body tell me why doesn’t my site appear in google search engine. my web site is [deleted by Lid]

    thank


  2. 10 September 2007 14:44

    My website has been live for 2 months now and I have had about 2,200 unique visitors to it mainly through google world advertising, but when I do a search for it from the google search engine it does not appear anywhere !

    Does it take so many months before you are recognised?

    Any pointers are very welcome

    Many thanks

    Richard James


  3. 17 March 2008 13:49

    I have a website http://www.fleurmarstravel.com and http://www.fleurmarstravel.biz. Since 2007

    However when I google it dies not exist.


  4. Lid
    17 March 2008 15:38

    Hi Fleurette

    Yep, you’re right. Google doesn’t really know about you. And it is not any one thing that makes this so, there are many reasons it doesn’t see you.

    First, you are part of the YTB travel network, so YTB is the one that shows up in search results (type “ytb travel network” into goog and you’ll see what I mean)

    Second, it seems that there are problems with YTB - some people seem to think it is spammy or scammy - see Goog search on YTB

    Additionally, both your biz and your com domain redirect to other pages that have no keywords, descriptions or any useful text. The majority of the site is in flash - and Goog can’t work that out. Use the lynx viewer (in article above) to see what Goog sees - not much but YTB

    That’s what I can see at first glance

    Good Luck!


  5. 18 April 2008 4:52

    My site does not show up when i type in the domain name or the keywords. It will show up if I type in the http://www. office-closet.com or my other site http://www.queensofhealth.com Unless people know the domain name they can not find me What can I do to fix this. I have submitted my site to search engines. Any help would be welcomed Thank you


  6. Lid
    18 April 2008 7:34

    Hi Bonnie

    Alas, your sites are not optimized at all. Just to start, you have:

    No sitemap,

    No meta data (keyword and description), (per page)

    No title element (per page)

    office closet has very little information

    both don’t seem to be updated often

    You could try typing in your sites to http://www.websitegrader.com to see what it offers up, but at a quick glance, you need to do some serious optimizing…

    Good luck


  7. 25 May 2008 7:16

    having similar problems with my website, can’t seem to find it when I search for magnets on stainless steel fridge…my site is http://www.choopaboard.com any guidance will be very helpful and appreciated! Thanks

    Mary


  8. 1 June 2008 22:26

    My website has been launched last month, but when I do a search for it from the google search engine it does not appear anywhere !

    Does it take so many months before you are recognised?


  9. 8 July 2008 6:40

    i have website http://www.earneasydollars.com/ . i buy a domain of my site thru blooger but my site is did not appear on google or any other search engine………………..i want to know…………..why??????//

  10. Lid
    8 July 2008 7:43

    @Mary - Howdy

    What a cool product you have - problem is, it is tricky to find - especially with Goog. Google does see you, but because you don’t have ‘magnets for stainless steel fridges’ *anywhere* Goog won’t associate your keywords with that.

    So a few things you could do.

    1. Get rid of the intro page - it serves no purpose, and from Goog’s point of view, it doesn’t tell them anything

    2. Your title is way too long, and doesn’t include any of your keywords. Also, it is the same for each page. It needs to be different for each page.

    Why not change the title of your main page from:

    “Choopa Board - suction cup to your refrigerator and magnetize your kids with artful fun!”

    to: (assuming you want the words you mentioned to work in Goog)

    “Magnets on Stainless Steel Fridges: Choopa Board”

    Your about page could be titled:

    “We make magnets for stainless steel fridges: About Choop Board”

    Your contact page could be titled:

    “Contact Choopa Board: Specializing in magnets for stainless fridges”

    3. Your description and keyword list for each page is way too long.

    Consider changing the description for each page (each page does contain something different) - keep it *UNDER* 160 characters

    Consider changing your keywords and cutting them down to under 10.

    At the moment you have too many and although keywords aren’t counted for much anymore, too many and it makes it look like you are spamming the system (also, some of them turn up twice - not good for SEO)

    This should be enough:

    magnets on stainless steel refrigerators,

    stainless steel fridge magnets,

    magnet board,

    fridge magnets,

    refrigerator magnets

    choopaboard,

    choopa board,

    4. You really have no content other than three pages. You’ll need to put up more information - more pages, with more content. Make it useful, don’t fill it up with keywords - if you can blog - do it. Consider using WordPress (on the same domain) and if you don’t want to blog regularly, try to do it at least 3 times a week for a couple of months, until you are noticed by the search engines - blogging may get you noticed in a week btw.

    5. Does anyone link to you? If you can get a few links in (legitimate - not from link farms) that are related in subject to you, all will be better. To help achieve this, back to blogging. You link to other sites, they may link back to you.

    6. Use alt tags for your images. Tell the search engines what they are and use different text for different images. If you can (and if you’re blogging), you can put a picture of a new magnet every day, write up a little story about it, make sure it includes your keywords (not too many remember - it needs to read well, not spammy). That way, you can have ONE page that talks about ‘frog magnets’ (and is titled that) etc.

    7. Add your site to various directories (good directories). I’ll be doing a post about this for the BlogWell Files in the next couple of weeks, detailing the better directories. Subscribe to my feed so you don’t miss it ;)

    8. You have both www and non www (see #3 below explained to Biswajit) - this is not good.

    I hope this is a little helpful and gives you a few ideas. I am really saddened that you’re not turning up because I really like your product - you just need to shove it into the search engines :)

    Good luck, and let me know if you get stuck again

    @Biswajit - wow - you’ve got tons of stuff there. Problem, like Mary, you have over optimized your site to the point where to search engines it looks spammy. Consider these things:

    1. Title: too long. Filled with keywords. Goog doesn’t like this. Better to have a different title for each page - and - if your service is location specific, include the location - get rid of the keywords, they don’t belong in the title.

    “Civil Engineering Consultant Services India” may be good for the home page - after that, look at each page and try and make it more specific to the topic of the page

    2. Keywords and Descriptions

    You have no meta data at all here. Include (for each page) a maximum of 10 keywords, and a description no longer than 160 characters.

    3. Decide whether you want Goog to know you as http://www.cecs.com or http://cecs.com - and the include a *permanent* redirect (301) - The fact that you are not doing this means that Goog has to split the rank between the two - better to use 301 and make Goog give ALL the Goog juice to one or the other.

    4. You are listed in DMOZ and Yahoo - which is great - but make sure you are also listed in other decent directories. If you are listed in directories that Goog thinks are fishy, that will tank your rankings as well.

    5. Like I mentioned to Mary, if you can blog, it will increase your chance of turning up in search results quicker. But don’t do it just for that sake - share some of your information with others (stay on topic) and watch Goog results get better. Link out, and let others link to you.

    To both of you, apologies for the delay in getting back - life gets in the way. :) but I hope this gives you some ideas on where to go.

    Additionally, we provide a telephone consult service (see our hire us page) - so if you want very specific information, and questions answered, we provide that service for a nominal fee.

    However, feel free to keep asking questions in the comments if you need help and I’ll do my best.

    Good luck to both of you, you really have great sites, and it is a shame to see them not ranking well.

  11. Lid
    8 July 2008 7:59

    @ Kamal

    Wow - lots of stuff on your site - but boy it looks a tad spammy (sorry)

    So. Much of what I’ve written above to Mary and Biswajit applies to you as well - read the comment I’ve left for them.

    No meta data - keyword or description

    Title - ‘Make Money Online’ - it is gonna be tough to get in the top results for this as there are many, many others doing it, and doing it far better (sorry again). You’ll need to up the ante if you want to succeed - this means better writing, better grammar, punctuation etc.

    Ads: - way too many - spammy looking - make sure they are all ‘nofollow’ otherwise you’ll be penalized by Goog

    Blog: Although you’re on Blogger, it doesn’t seem as if it is a blog - is it?

    Sorry - really can’t help much more than that.

    Good luck! It’s a difficult niche to hit.

  12. 28 July 2008 9:15

    Hey there -

    I appreciate the tips given to Mary and Biswajit. I seem to have the same problem with my site: http://www.ravensbark.com. It has been active for several months and does not exist on the search engines. I think I have thought through and implemented the tips you have previously given, but it still doesn’t seem to help. Any comments are appreciated. Thank you.

    Bob.


  13. 30 July 2008 14:05

    Hi,

    I have been reading your comments and suggestions and I think they are great and wondered if you could help me too. I have the same problem as other - i have added my URL to all major search engines inc google…and yet I cant find me!!! SOB.

    I have added meta tags, strategically placed keywords, and checked how it looks with website grader. Im a little naive at all the weird and wonderful googley things and would appreciate any help for my website http://www.tiemeup.org.uk (eherm..excuse content!)

  14. Lid
    6 August 2008 12:34

    Belle, sorry it has taken a while to reply.

    Thank you for your kind words about BlogWell.

    Let’s take a look at your site:

    Google does know you exist, see here:

    Google results

    But, while you have added meta tags, keywords etc, it seems you’ve gone a little overboard on the home page, but not created meta tags etc for the other pages.

    As each page is different, give it a different title, description and keywords

    Try to keep your title under 60 characters, your description under 160 characters and your keywords no more than ten on every page.

    Also, you seem to have quite a few H1 tags; really each page should have only one, the rest should be H2 - H6, depending on importance

    The other thing you should consider is to decide whether you want to be www or non www because at the moment, your site shows up as both. You want to direct any Google juice to one, so decide which one, create a permanent redirect from the one you don’t want to the one you want, and that will help too.

    Finally, you don’t seem to have many (if any) links going to your site. You’ll need to think of ways to get links, and quickly. They are important across all search engines.

    I hope this helps a little :)

    If you need help rewriting title/description/tags, take a look at our services page, it is something we provide on a freelance basis.

    Good luck!

  15. Lid
    6 August 2008 16:28

    Rob,

    Apologies for taking so long to reply; hope this will still be useful.

    What a wonderful, wickedly great product you have!

    Now, about that SEO…

    Lots to say, don’t know where to start so I hope you’re ready for some work!

    First thing:

    Google does know you exist, but you have no ranking. Don’t forget, oftentimes Google wont start ranking you for 6 - 12 months if you have a static website, so it is not unusual.

    Not only does Google look at how long the site has been up, it also looks at how long before the registration runs out (it helps them determine whether you are here for the long run). If your domain registration is set to run out in less than 12 months, renew it now so that it doesn’t run out for 2 - 5 years.

    Get links

    One of the best ways to rank is to get links, that is, get other [relevant] sites to link to you. One way to do this is to place ads (even 125 x 125 ads) on various blogs that are relevant to gourmet foods. Prices vary, but are usually fairly reasonable $30 - $100 for a month, depending on the blog. You could start with one or two and see if it brings in any visitors/traffic; it will bring in a link.

    Submit your site to the Yahoo Directory. It costs about 299/year, but it does help get you into the results.

    Alternatively, have you considered blogging? Lots of small businesses are turning to blogging as it provides fresh content regularly, something that Google particularly like. Additionally, you may find over time people will start linking to the blog, which in turn is connected to the site, assuming you set it up that way in the first place. While it is not too difficult or expensive to set up, it does require commitment on your part. Think about it.

    You have included titles on some pages, not others, but I am glad to see you name them differently. Create titles for every page. Make sure you don’t exceed 60 characters (including spaces)

    Meta Description: For your homepage, your description is waaaaaaaay too long :) cut it down to under 160 characters, then write a description for each page, make sure it’s different - after all, each page is different.

    Keywords - the average is about 10, you’ve got a few more, but it shouldn’t really cause a problem. Do it for every page, try and mix them up so that your keywords are not the same across your site.

    A word about keywords. The more specific you can be, the better. I’ve noticed you are quite specific with descriptions, but the keywords not as much.

    Now a question for you:

    For your company to turn up in search results, what do you think people need to type into the Google search box?

  16. 13 August 2008 5:51

    I have only just created and published a website, http://www.mikeperryfitness.co.uk I have summitted it to google and still don’t have a listing. Is there something else that I could try, thank you lid


  17. Lid
    14 August 2008 9:31

    Hi Mike,

    Oftentimes, Google needs weeks if not months to recognize your site exists. So if it hasn’t been that long, don’t fret yet.

    However, there is much you can do to your site to improve your chances of the Googlebot noticing you.

    If you take a look at my reply to Rob (above) you’ll see that many of the things he has not done, need to be done by you as well.

    So, here’s a start:

    While you do have titles for your pages, you have no description or keywords for any of the pages. This is important. You need to describe each page, in under 160 ish characters to Goog. You also need to include keywords for each page - no more than 10 (ish). Because each page is different, you’ll need different descriptions, different keywords.

    Take a look at this post on BlogWell:

    How to have a clean result in Google

    Also, you have the ULR http://www.mikeperryfitness.co.uk as well as http://mikeperry.co.uk - you’ll need to determine which one you want to use, and then redirect the other [permanently] to the one you will be using - otherwise, you’ll just confuse the Googlebot ;)

    Also, while you do have titles on your pages, the page slug is fairly useless from a search engine point of view. For instance, you have id8.html for the slug on your ’sports and body massage’ page - and that doesn’t really tell Goog anything.

    http://www.mikeperryfitness.co.uk/id8.html

    Better, would be to change it to http://www.mikeperyyfitness.co.uk/sports-and-body-massage.html so that it is in the URL as well as the title. You should do the same for each page.

    You haven’t been crawled by Google yet that I can see; one way to get them to notice you is to have links to your site from other, related and relevant sites.

    The other thing I would consider is paying to be included in the Yahoo directory (it’s about $299 p/year). That usually helps. Also, try getting included in DMOZ. DMOZ doesn’t charge, but then, they don’t guarantee a listing either.

    https://ecom.yahoo.com/dir/submit/intro/

    http://www.dmoz.org/

    Hope this helps a little. Best of luck, and let me know how you go.


  18. 19 October 2008 14:53

    Thanks for your advice Lid, I will crack on with making those improvements when I get a few minutes. I have managed to get listed on google and appear on page one on Yahoo. I hadn’t even realise that I hadn’t got any keywords. It’s always the simple things that I over look! Thank you very much for taking time out and your good advice. I have added alot more content and will continue to make the improvementd you suggest.


  19. 12 November 2008 4:03

    Hi I wondered if you could help a complete novice who doesnt seem to be recognized on any search unless the person puts http//:bestcatflaps.co.uk. I have done the website myself but am lost - what can I do? I need to appear on google/yahoo…etc searches to get anywhere - there are bigger competition out there who seem to appear everywhere. If you have time - please help me? Please give me the dummy guide? Thanks Emma x

1 trackback:

  1. Search Engine Friendly Design « BlogWell 1 July 2007

Write a comment: