How can I check if my site has been crawled?

[zannix]

Hi.

I'm interested if there are any online free ways/services to check if my site was crawled by google, and when it last occured?

I need this because the way I'm doing it now is changing the site description by one word and then checking the google's listings to see if it's updated the description. Then I know it's crawled.


But I'm interested, there surely must a way around this? Something quicker.

[renukoot]

Hi,

I have some site address at my site. Hope that may help you.

Search Engine Optimization (SEO) - Better SEM & SEO Optimization Tips Tutorials

Thanks & Regards
Abhishek Srivastava
(www.needtruehelp.com - 1st Shopper Approved website selling Traffic)

[Steve Faber]

look in your site logs at your hosting account. You'll be able to see if any of the search engine bots have crawled your site, and when they did so. If you're using CPanel for your hosting interface, it's really easy. Just go to the logs section and use awstats. On the left, after entering awstats, look under the "who" section under robots/spiders visitors. It's al right there.

[SmartHealthShop]

you can use Google analytics or Engineseeker to see all the IPs browsing your site. check the IP and you can see its location (look for CA IPs and also WA state IPs)

[hyderkhan]

Signup for Google Webmaster Tools. You will be able to login and visibly see the date and time of when was the last time Google crawled your website, as well as a whole host of other useful statistics that Google maintains (eg- backlinks, keywords used to search your site, your site's ranking for certain keywords, etc.)

[Christian Little]

Signup for Google Webmaster Tools (https://www.google.com/webmasters/tools/). It takes a few minutes to setup for your site, but it provides you with a ton of useful data including how/when Google crawls your site. It even provides you with a list of problems that Google has found when crawling your site, which is very handy.

EDIT: hyderkhan beat me to it by a few seconds lol.

[zannix]

Thank you guys! Great help!

BTW - do you know any method which kind of "let's google know that it should crawl my website", or do I just wait for it to happen?

I mean I'm indexed already, but I'm building backlinks, and I want to monitor progress in listing rankings.

[hyderkhan]

Originally Posted by zannix

Thank you guys! Great help!

BTW - do you know any method which kind of "let's google know that it should crawl my website", or do I just wait for it to happen?

I mean I'm indexed already, but I'm building backlinks, and I want to monitor progress in listing rankings.

One way is to make sure that you have an updated sitemap.xml hosted on your website.

And then log onto Google Webmaster Tools and select the option to have Google download an updated sitemap.xml from your website. (Not sure of the exact phrasing, but the option is there.)

This is one way of signalling to Google "hey I just updated my website... please crawl me!" But Google will still take its own good old time to do so. But it least you'll be queued up.

[Andy Yusuf]

use google webmaster tools mate...

[dburk]

Originally Posted by zannix

Thank you guys! Great help!

BTW - do you know any method which kind of "let's google know that it should crawl my website", or do I just wait for it to happen?

I mean I'm indexed already, but I'm building backlinks, and I want to monitor progress in listing rankings.

Hi zannix,

There is a special protocol designed specifically for this purpose, it's called a XML-RPC ping.

[Christian Little]

Google Webmaster Tools will tell you how often Googlebot is crawling your site. You can also ask them to crawl it more frequently in the interface. They give you a pretty little chart that shows you these details, here's for one of my sites:



That shows me that Googlebot is crawling 14 pages/day on my site usually, but it had a big spike in February.

[RogerBell]

you might try wordpress-crawl-rate-tracker you'll have to google it I can't post links

[zannix]

Thank you guys.

I suppose pinging a site at pingomatic won't make google go crawl?

[dburk]

Originally Posted by zannix

Thank you guys.

I suppose pinging a site at pingomatic won't make google go crawl?

Technically you are not pinging a "site", you are pinging a URL, and pinging with Ping-O-Matic will almost always send the Googlebots a calling. You can also use Twitter and it will often work even faster than pinging.