7 {{ upvoteCount | shortNum }}
7 {{ upvoteCount | shortNum }}

Can someone tell what this crawl test result means..site was deindexed

by jlady
5 replies
I just did a crawl test in webmaster on a site that was deindexed and got the following results...does anyone know what it means???

This is in my robots.txt...

User-agent: *
Disallow:

This is what came up after I asked google/webmaster to do a crawl test.....

Allowed by line 2: Disallow:
Detected as a directory; specific files may have different restrictions
#crawl #deindexed #meanssite #result
  • Profile picture of the author Capitalist_Pig
    It means that your site is open to all bots, and that individual files could have different permissions in the meta tags.
    {{ DiscussionBoard.errors[2839087].message }}
  • Profile picture of the author jlady
    So from the crawl response there is nothing there that would indicate why I am deindexed.
    {{ DiscussionBoard.errors[2839459].message }}
    • Profile picture of the author JohnMcCabe
      It indicates that you were de-indexed for some reason other than a restriction in your robots.txt file.

      Any bot can access your site, at least on the top level. You might want to check your site statistics under user-agents. Most of them will be some flavor of browser, but the one you want to see here is googlebot. If you see that, your site is being visited and left out of the index for some reason.
      {{ DiscussionBoard.errors[2839738].message }}
      • Profile picture of the author jlady
        Not sure what you mean by user agents or where to find it, but I did

        fetch as googlebot and it was successful.

        Originally Posted by JohnMcCabe View Post

        It indicates that you were de-indexed for some reason other than a restriction in your robots.txt file.

        Any bot can access your site, at least on the top level. You might want to check your site statistics under user-agents. Most of them will be some flavor of browser, but the one you want to see here is googlebot. If you see that, your site is being visited and left out of the index for some reason.
        {{ DiscussionBoard.errors[2840159].message }}
        • Profile picture of the author JohnMcCabe
          Originally Posted by jlady View Post

          Not sure what you mean by user agents or where to find it, but I did

          fetch as googlebot and it was successful.
          User agents are the programs visitors use to "see" your content. Human visitors will normally use a web browser like Internet Explorer, Firefox, Chrome, etc.

          Search spiders and other computer programs (grouped together as "bots") go through your web pages by scanning the source code - What you see if you right-click on a web page and select "View Source".

          Googlebot is the user agent Google uses to access your site and spider your pages.

          If your web host uses Cpanel, look under statistics. Most hosts offer one or two different tools for examining your server logs - files which track requests for data by browsers, what's returned, etc..

          One of the standard categories in those tools is User-agents. You get a list of the programs used to request your web pages, graphics, and such.

          If you look at that section, and find Googlebot, Yahoo Slurp and (just forgot the MSN spider's name), it means that those search engines are accessing your content.

          Webmaster tools was able to fetch your home page using googlebot.

          That means that, if your site is de-indexed, the problem is not spider access. Beyond that, it doesn't tell you much...
          {{ DiscussionBoard.errors[2840231].message }}

Trending Topics