How to Increase Google Crawl Rate

Google is king when it comes to the search engines, which is why its pretty darned important to get them to index your pages quickly & efficiently. So I’ve decided to share some of my experience with you. So you can improve your website crawl rate, get those pages indexed quicker & and increase your website traffic.

1. Frequent Content

This sounds like an obvious one, but from my experience people just don’t do it enough. Google loves content, its text it sees, not much else really. So the more content you have, on a regular basis, it then knows to visit you more frequently, to get the latest stuff. Unless you’re a obsessive blogger, and remember this applies to websites which aren’t blogs too. Aim to add new content, once per week. This should over time, and I mean a few months, get google returning every week to index your new pages. 

I do admit this can be tricky, especially if you’re website is a very specific topic, as its not easy to add content that frequent. A few suggestions for this is add news about your product, make sure its unique as this will carry more weight. Another idea is to add a corporate blog, this will give your customers a way to get to know your company personally, and it’s far easier to write generate more content this way, but do not just re-publish press releases…

2. Update Every Page

From the title, this one sounds stupid, but this is a great ‘trick’. Google is stupid, just like the other search engines. A lot of websites use includes as a common element in their pages, like a header or footer. Add a nice strap-line describing your website, and change this frequently and regularly. Using the spoon example you could put something like: “Spoonsite.co.uk provides information on spoons and other cutlery” – when you update this weekly, try and increase or decrease it by at least 30 characters, just so google can then see the page size change, and believe all your pages have been update. It does increase the google crawl rate.

3. Good Linking

Internal linking is as important as external links. If you have good linking, then Google can navigate all your your pages easily, increasing the chances of your pages being indexed. External linking is also important, if someone links to you, when they are crawled, you have a pretty good chance the bot will follow that link through to your site, and hey presto, you’re being crawled again.

4. Just check the obvious

If you’ve go a webmaster tools account, make sure you don’t have your crawl rate set to slower, as this wont be necessary for your website usually, only if you have problems. Also check out your robots.txt – that you haven’t actually blocked some bots from visiting. You will be surprised how many people do this.

Ok here are a few tips, enjoy and give them ago.

Thanks

Wuup Team

Tags: , , , , , , , , , , , ,

Author:Alan Hamlyn

-- Alan Hamlyn Founder of Wuup
  • http://www.insurance4all.co.uk Anna

    Great ideas to help motivate me as I’ve found over half my website hasn’t been indexed by google. I’m going to check that I haven’t set Google to crawl slowly. Anyone who doesn’t have google analytics should go and get them as you can learn a lot on how google is viewing and crawling your site.

  • Amber
  • http://www.farbige-kontaktlinsen.ch/ farbige kontaktlinsen

    After reading this usefull article how to use twitter. I even see some sens in twittering. Thx a lot.

  • http://www.federico-tomassetti.it Federico Tomassetti

    I am doing as you advised but until now my site is not yet been crawled by Google… :(

  • http://www.wuup.co.uk Alan Hamlyn

    Hi Fredrico, have you got any other websites which have been indexed by google? Also if that is the site in question you have linked to from you comment, then I can almost certainly say it will be crawled soon from Wuup -> Yours :)

  • http://www.wuup.co.uk Alan Hamlyn

    Upon further looking, it already has :) look at http://www.google.co.uk/search?hl=en&safe=off&q=site%3Ahttp%3A%2F%2Fwww.federico-tomassetti.it%2F&btnG=Search&meta=&aq=f&oq= – the rest of the pages will be indexed soon :)

  • http://www.stop-smoking-with-cigarrest.com CigArrest

    Just wanted to give you a quick thanks. And let you and everyone know that I took your advice and set my crawl rate on my webmaster tools account via google to faster. Really dont understand why it was on slower to start with. But thanks for your advice. My CigArrest blog is already being indexed more than ever. Thanks again

  • http://www.carpetbagging.co.uk David

    I changed the internal links a couple of months ago and as you say appears to be very important as not only did i find pages getting indexed more often but also a few weeks later many were appearing a lot higher in the search results.

  • http://www.wuup.co.uk Alan Hamlyn

    Hi David, that is great news, glad that the advice helped. Yes internal linking is absolutely a ranking factor hence you saying that results were appearing higher.

    All the best.

    Alan

  • http://www.zenithuk.co.uk SOEX

    Really very informative post, thanks for this.

  • http://www.chauffeurlinks.com Chauffeurlinks

    Thanks Alan Hamlyn for this useful post. No doubt, Content is king and update the content of website regular bases can increase the chance of immediately crawl rate by Google.

    Really, I like this article too much and going to bookmarking this page.

  • http://www.learntotradethemarket.com price action

    Regular content and , plenty of quality action on the site, and keeping the prices of producing that copywriting down, is the best strategy to ensure fast crawling, especially if its in competetive niches where there is plenty of action, and most price of traffic is a high cost, the answer is genuin, regular content and good site links.

  • http://www.wuup.co.uk Alan Hamlyn

    really good tips there, thanks for stopping by :)

    alan

  • http://www.promotionalcodes4u.co.uk Sue

    Content, content and content, all the things mentioned above are great but if one can add a new page each day eventually the google bots will appear more and more frequently. If time is short can sometimes be worth writing a few pages but only publishing one a day until google has got the message that the site is being updated consistently.
    As others have said the site linking is important to allow the bots to discover the new content quickly.

  • http://www.onlinestoresreview.co.uk Peter

    Any tips for yahoo and bing as they are the search engines that i find seem to take an eternity to index an entire site?