18 replies
  • SEO
  • |
I set up a site almost 1 week ago. I have pinged the url and submitted the site to a few social bookmarking sites and also created a sitemap etc. The site has still not been indexed by Google. Previously almost all my sites have been indexed within 24 - 48 hrs.

Is this still fairly common or am I doing something wrong?
#indexed #site
  • Profile picture of the author yaniman
    Please check your robots.txt and make sure your site map is indexed in Google webmastertools.
    The best way to index your website is get a solid backlink from an authority website in your niche.
    {{ DiscussionBoard.errors[2534679].message }}
  • Profile picture of the author tjm2788
    I've checked the robots.txt in Webmaster toold and the sitemap is indexed. I could do more bakclinks but surely at this point just to get indexed shouldnt require this? I have links from YouTube and a few social bookmarking sites too. Usually my sites get indexed without doing anything within a few days im finding this strange!
    {{ DiscussionBoard.errors[2534716].message }}
  • Profile picture of the author yaniman
    It seems strange I indexed my new website in Google by just digg it. May be your server is slow or the server IP is blacklisted (just a doubt only). Please try to promote your website in more social bookmarking sites , it can attract spiders to your website.
    {{ DiscussionBoard.errors[2534731].message }}
  • Profile picture of the author tjm2788
    Ahh I think I may have found the problem in the robots.txt file: my other sites all have

    Allow: /

    but this site has

    User-agent: *
    Disallow:

    If that is the problem how come it is like that my default? I have never had to edit this file before!
    {{ DiscussionBoard.errors[2534749].message }}
    • Profile picture of the author Google.me
      Originally Posted by tjm2788 View Post

      Ahh I think I may have found the problem in the robots.txt file: my other sites all have

      Allow: /

      but this site has

      User-agent: *
      Disallow:

      If that is the problem how come it is like that my default? I have never had to edit this file before!
      Disallow:/ means dnt go
      Disallow: can go

      Allow: / Ive tryd it and it doesnt work
      {{ DiscussionBoard.errors[2534776].message }}
    • Profile picture of the author TristanPerry
      Originally Posted by tjm2788 View Post

      Ahh I think I may have found the problem in the robots.txt file: my other sites all have

      Allow: /

      but this site has

      User-agent: *
      Disallow:

      If that is the problem how come it is like that my default? I have never had to edit this file before!
      User-agent: *
      Disallow:

      is fine. That's what it should be. Has the Google Bot been on your site? Check your traffic stats to see. Sometimes they're just a little slow for no real reason.
      Signature
      Plagiarism Guard - Protect Against Content Theft
      {{ DiscussionBoard.errors[2534787].message }}
  • Profile picture of the author tjm2788
    So to allow google and everything else it should read:

    User-agent: *
    Disallow:

    Is that correct?
    {{ DiscussionBoard.errors[2534785].message }}
    • Profile picture of the author Google.me
      Originally Posted by tjm2788 View Post

      So to allow google and everything else it should read:

      User-agent: *
      Disallow:

      Is that correct?
      ill just use

      Disallow:/ for blocking stuff and thats it. The bots will follow the rest well I have no probs http 200 sweet
      {{ DiscussionBoard.errors[2534795].message }}
  • Profile picture of the author tjm2788
    Ok I have changed the robots.txt file. I have checked my stats my host stats arent compiled yet as its just been a week. Can I check to see if the googlebot has visited with Google analytics? I cant seem to find where I can check...
    {{ DiscussionBoard.errors[2534808].message }}
    • Profile picture of the author TristanPerry
      Originally Posted by tjm2788 View Post

      Ok I have changed the robots.txt file. I have checked my stats my host stats arent compiled yet as its just been a week. Can I check to see if the googlebot has visited with Google analytics? I cant seem to find where I can check...
      You didn't need to change the robots.txt file; it was exactly as it should be. "Disallow: " is the correct way to say that the site is open to bots

      The Google Bot doesn't show up in Google Analytics, no.

      Have you signed up to Webmaster Tools? Since "Crawl Stats" under "Diagnostics" show whether the Google Bot has visited your site And also check "Crawl Errors" to see whether Google is reporting any problems with your site.
      Signature
      Plagiarism Guard - Protect Against Content Theft
      {{ DiscussionBoard.errors[2534835].message }}
      • Profile picture of the author james7
        I think some site take more time to indexed. But you continue on social bookmarking, article submission which makes your site indexing more quickly.
        {{ DiscussionBoard.errors[2534894].message }}
  • Profile picture of the author Google.me
    google webmaster console and do all test in crawl access. Give it a few minutes or hours to see the results
    {{ DiscussionBoard.errors[2534821].message }}
  • Profile picture of the author tjm2788
    Thanks... I checked under webmaster tools and it is not reporting any errors but showing activity:

    Pages crawled per day:
    High: 8
    Average: 5
    Low: 2
    {{ DiscussionBoard.errors[2534858].message }}
    • Profile picture of the author Google.me
      Originally Posted by tjm2788 View Post

      Thanks... I checked under webmaster tools and it is not reporting any errors but showing activity:

      Pages crawled per day:
      High: 8
      Average: 5
      Low: 2
      just wait for them to download your robots.txt
      {{ DiscussionBoard.errors[2534882].message }}
  • Profile picture of the author TristanPerry
    That's a good sign It probably just means it's going a little slow for some reason.

    It can act a little weirdly although if you are getting no crawl errors and the Google bot is visiting your site, you should be okay.
    Signature
    Plagiarism Guard - Protect Against Content Theft
    {{ DiscussionBoard.errors[2534868].message }}
  • Profile picture of the author tjm2788
    Ok then cheers guys, I know it can take time just wanted to make sure there wasnt a specific reason for not indexing as generally my sites appear really fast. I will just give it time.

    Thanks for all your help!
    {{ DiscussionBoard.errors[2534901].message }}
  • Originally Posted by tjm2788 View Post

    I set up a site almost 1 week ago. I have pinged the url and submitted the site to a few social bookmarking sites and also created a sitemap etc. The site has still not been indexed by Google. Previously almost all my sites have been indexed within 24 - 48 hrs.

    Is this still fairly common or am I doing something wrong?
    You can submit all the sitemap.xml's that you want, ping the search engines all you want, but Google and the other engines are going to crawl and "maybe" index your site whenever they feel like it. They typically aren't in that big of a hurry to crawl new sites unless they start stumbling onto links to your new site when crawling.

    Honestly, I've never understood why people put forth so much effort building/submitting sitemaps and pinging to get indexed. Typically, their site is not going to rank for squat unless they have built links.

    So you get indexed. It has done you no good other than to make you feel good that you can search for site:yourdomain.tld and see your site there. But no one else will be able to find it unless maybe your pages show on page 1 for a few days due to the Google honeymoon.

    You're much better off spending that same time and effort you spent getting indexed to simply build links and let the engines discover your site the old fashioned way.
    {{ DiscussionBoard.errors[2534920].message }}

Trending Topics