17 replies
  • WEB DESIGN
  • |
I know this will sound stupid to most of you, but how do I know when my site has been crawled by google and others?
  • Profile picture of the author Talkreal
    Hi goin4it,
    This is not a stupid question at all. There are numerous ways to tell if your site has been crawled. Do you have a web hosting account? If so, view your web server stats. In cPanel, for example, go to Logs --> Awstats. It will tell you who has visited your site, including the robots. It will also let you when they last visited.

    You should also sign up for Google Analytics and Webmaster Tools. You will have powerful site monitoring with those two in tandem, and they're free.
    {{ DiscussionBoard.errors[3619497].message }}
    • Profile picture of the author Qualibid
      You should definitely sign up for Google webmaster tools.
      It's a must-do step for everyone who cares about Google indexing.


      Originally Posted by Talkreal View Post

      Hi goin4it,
      This is not a stupid question at all. There are numerous ways to tell if your site has been crawled. Do you have a web hosting account? If so, view your web server stats. In cPanel, for example, go to Logs --> Awstats. It will tell you who has visited your site, including the robots. It will also let you when they last visited.

      You should also sign up for Google Analytics and Webmaster Tools. You will have powerful site monitoring with those two in tandem, and they're free.
      {{ DiscussionBoard.errors[3620758].message }}
      • Profile picture of the author goin4it
        Thank You all for the very helpful input! Got analytics and Google Web Master Tools. Now I just need to learn how to use them Thanks again!
        {{ DiscussionBoard.errors[3623073].message }}
  • Profile picture of the author haymanpl
    install a free plugin called statpress or wordpress.com stats
    {{ DiscussionBoard.errors[3620344].message }}
    • Profile picture of the author JRCarson
      If you put your website in quotes, "http://www.mywebsite.com" and put it in the Google search bar, it will show you which pages Google has indexed on your site.
      {{ DiscussionBoard.errors[3620362].message }}
  • Profile picture of the author kjhosein
    Yup, @qualibid beat me to it on the Webmaster Tools front, but here's one more tip: ensure that you generate a sitemap (sitemap.xml). If you do, it'll make Google's job of crawling and indexing your site that much easier. It's sort of like handing them a phone book of names, so they don't have to go knock on everyone's door.

    Webmaster Tools also allows you to submit a sitemap, and will tell you if it's valid.

    HTH!
    Signature
    <!--PM me for a quicker reply. Thx!-->
    {{ DiscussionBoard.errors[3623226].message }}
    • Profile picture of the author marcellarhughes
      There are a lot of free tools to use, to identify your site was already crawled by google. Use google webmaster tools that's free tools good luck!

      __________________________________________________ ___________
      CHI Premier - Premier IT Solutions
      Web Designs
      Web Hosting
      UK IT Consultancy
      {{ DiscussionBoard.errors[3624503].message }}
    • Profile picture of the author goin4it
      Originally Posted by kjhosein View Post

      Yup, @qualibid beat me to it on the Webmaster Tools front, but here's one more tip: ensure that you generate a sitemap (sitemap.xml). If you do, it'll make Google's job of crawling and indexing your site that much easier. It's sort of like handing them a phone book of names, so they don't have to go knock on everyone's door.

      Webmaster Tools also allows you to submit a sitemap, and will tell you if it's valid.

      HTH!

      kjhosein....thank you for this info! I tried to submit my site map and I got a response that says "Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead". I definitely don’t know what to do from here...any suggestions?? Thanks everyone for your help!
      {{ DiscussionBoard.errors[3627194].message }}
  • Profile picture of the author Microsys
    You need to create an XML sitemap and submit that instead. Here's a series of xml sitemap tutorials taking you from creating to submitting.
    {{ DiscussionBoard.errors[3628480].message }}
  • Profile picture of the author kjhosein
    Yup, what @sitemapper said. If you're running WordPress or other popular CMS, there are plugins that automatically generate sitemaps periodically.

    You can even manually generate a simple sitemap if your site doesn't have a lot of pages. Here is Google's how to on the subject, and you can take their word for it, b/c they're the ones that are processing the file, so they'll know best what they want!
    Signature
    <!--PM me for a quicker reply. Thx!-->
    {{ DiscussionBoard.errors[3629846].message }}
  • Profile picture of the author staciejortega
    i think.. when it is already cached by google.. but im not sure

    you can use the google operator cache

    for ex: cache:www.warriorforum.com

    {{ DiscussionBoard.errors[3630418].message }}
  • Profile picture of the author haymanpl
    If you install the Stats plugin it will tell you exactly which spiders crawl your site
    {{ DiscussionBoard.errors[3769920].message }}
    • Profile picture of the author airallineed
      Use Google Webmaster Tools, it will tell you when google visited your site
      Signature

      selling emailtemplates.info
      PM me if you are interested!

      {{ DiscussionBoard.errors[3770471].message }}
  • Profile picture of the author Alex Peltonen
    If you want to see when your site was crawled, type cache:<your url> or select Cached snapshot of page from Google toolbar when you are on the website. Once the site is re-crawled and Googlebot indexes the updated content, it will be shown in the search results.
    {{ DiscussionBoard.errors[3777236].message }}
  • Profile picture of the author Money maker guy
    Use google
    Signature
    Hi Guys!
    This is my first page: http://hubpages.com/t/22e791
    Take a look....!!
    {{ DiscussionBoard.errors[3781659].message }}
  • Profile picture of the author express1431
    [DELETED]
    {{ DiscussionBoard.errors[3783437].message }}
    • Profile picture of the author Istvan Horvath
      Originally Posted by express1431 View Post

      Thanks for your great information friends........
      Stop your nonsense one line posts before they kick you out!
      Signature

      {{ DiscussionBoard.errors[3783445].message }}
  • Profile picture of the author cashxpert
    I sell a software called searchengine detector from this webpage Viral PLR Software

    It comes with plr.
    Signature

    Henrik Wanner-Olsen

    Free Traffic On Demand.
    http://www.rotatortrafic.com

    {{ DiscussionBoard.errors[3783681].message }}

Trending Topics