Google crawler bot help ?

by 6 replies
7
Hello
I need some help from you to assist me. I installed wordpress theme to my domain but unfortunate, Google bot has been blocked to crawl on site in privacy option so my site gone to out of cache in Google and all major search engines, now i made changed in wordpress theme and unblock the Google bot crawlers. so this changes would be automated updated in webmaster tool because i find there also site crawler access is blocked in robot file. i have made changes this few hours ago today or i have to made manually correction in robot text file to access Google my site..

Thanks for your valuable opinions
#programming #bot #crawler #google
  • Install any of the popular seo plugins for wordpress. Then configure all the settings, create a sitemap and send it to google. Should be good.
  • Using your web browser, type in:

    www.yourdomain.com/robots.txt (obviously change yourdomain to your url)

    and see what your robots.txt file is saying. Make sure all is good before telling BigG to come back again.

    Bruce
  • All is ok there...
    User-agent: *
    Disallow:
    in robots.txt file but still Google webmaster tool is showing 16 urls restricted
    they would be automatically updated in webmaster tool ?
  • Some themes block /archive/ and /tag/ URLs. You might check to see if that's the case.
  • where i should check that blocked archive/ and /tag/ URLs
    • [1] reply
    • user agent should be allowed in robot.txt file . Generate it again from your webmaster tools and upload again it to your web server.

      aprt from this you can create your sitemap from this site Create your Google Sitemap Online - XML Sitemaps Generator and submit sitemap url to your sitemap section in webmaster tool . just like yourdomain.com/sitemap.xml
Join the discussion

Next Topics on Trending Feed

  • 7

    Google crawler bot help ?

    Hello I need some help from you to assist me. I installed wordpress theme to my domain but unfortunate, Google bot has been blocked to crawl on site in privacy option so my site gone to out of cache in Google and all major search engines, now i made changed in wordpress theme and unblock the Google bot crawlers. so this changes would be automated updated in webmaster tool because i find there also site crawler access is blocked in robot file. i have made changes this few hours ago today or i have to made manually correction in robot text file to access Google my site..