WtF Why My SIte Didnt Crawls

2 replies
  • SEO
  • |
User-agent: * Crawl-delay: 1 Disallow: /wp-content/plugins/ Disallow: /wp-admin/


What is this means? Crawl Delay?
#crawls #site #wtf
  • Profile picture of the author thebigweb
    Originally Posted by TechMate001 View Post

    User-agent: * Crawl-delay: 1 Disallow: /wp-content/plugins/ Disallow: /wp-admin/


    What is this means? Crawl Delay?
    Where are you getting that message?
    {{ DiscussionBoard.errors[10495248].message }}
  • Profile picture of the author nettiapina
    Why don't you just copy these to the Google search box? Here's what Wikipedia has to say about robots.txt.

    The robots.txt doesn't seem to be the reason why the site wasn't crawled, but the changes of someone being right about the crawl status are something like 10/90 against them. We need to see the site in order to help.
    Signature
    Links in signature will not help your SEO. Not on this site, and not on any other forum.
    Who told me this? An ex Google web spam engineer.

    What's your excuse?
    {{ DiscussionBoard.errors[10495413].message }}

Trending Topics