9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

How often do sites get crawled?

by DavidO
8 replies
  • SEO
    • |
I was under the impression that the more often you update you site, the more often the spiders will come calling.

Now I'm totally confused... I've not only increased the frequency of adding new pages but I've also installed an RSS feed and I ping and bookmark the new pages. I also update my sitemap each time.

But the article I published a week ago is still not showing up in Google. In the past I've had a new page indexed within hours and I wasn't even updating my site regularly.

Now I have no idea what their system is.
#crawled #sites
  • Profile picture of the author Solidsnake
    Banned
    No one could tell when google robot crawl your site... But I think it's daily..
    {{ DiscussionBoard.errors[1106496].message }}
  • Profile picture of the author vivagogo
    ping them as well, but don't do it too much, especially with blog system
    {{ DiscussionBoard.errors[1112208].message }}
  • Profile picture of the author Kappa
    you can go to google webmaster tools, they give you a crawl frequency rate after you verify your website, I think with mine it's daily.
    Signature

    {{ DiscussionBoard.errors[1113184].message }}
  • Profile picture of the author TristanPerry
    There's no set number. For larger sites it'd be once every second (or thereabouts), for very small static ones it may be weekly or less.

    I once had a site which I *think* had a minor penalty placed on it. I completely re-designed the site and slapped up all unique content and all, and it taken Google a week to realise I had changed my site. It then taken a few more days before it visited any of my new pages (even after submitting it to webmaster tools and all)
    Signature
    Plagiarism Guard - Protect Against Content Theft
    {{ DiscussionBoard.errors[1113220].message }}
  • Profile picture of the author articlemaster
    Originally Posted by DavidO View Post

    I was under the impression that the more often you update you site, the more often the spiders will come calling.

    Now I'm totally confused... I've not only increased the frequency of adding new pages but I've also installed an RSS feed and I ping and bookmark the new pages. I also update my sitemap each time.

    But the article I published a week ago is still not showing up in Google. In the past I've had a new page indexed within hours and I wasn't even updating my site regularly.

    Now I have no idea what their system is.
    Generally they do it on a regular basis. Go to your webmasters tool and check the crawler frequency.You can see which page is scrolled and indexed.
    Signature
    Submit Articles
    {{ DiscussionBoard.errors[1113245].message }}
  • Profile picture of the author giliav
    Google tens to tweak the re-visit rate based on your content update frequency.
    I've seen sites I own in which the crawler comes in 1-2 times a day (i.e. social bookmarking sites) and others in which it will come around once a week or two (i.e static content sites).
    Signature

    Thanks,
    Gilad.
    Signatures

    {{ DiscussionBoard.errors[1113290].message }}
  • Profile picture of the author Jimerson Farveez
    Here is the solution: Get more backlinks, Daily try to get at least 20-25 backlinks, After 2-3 weeks, you will see, google had visited all your pages.
    Signature
    Search Engine Optimization Services in Delray Beach, FL
    {{ DiscussionBoard.errors[1113297].message }}

Trending Topics