Privacy - do not index - doesn't work

9 replies
  • SEO
  • |
Beware. Just because you check the Privacy setting (Do not index) in Wordpress, Google WILL crawl your site and WILL rank it. I had a site that was chedked from the moment it was created and it still got ranked even though it is still checked. Ironically it ended up on the FIRST PAGE. But the description on the search page says " A description for this result is not available because of this site's robots.txt - learn more."
#index #privacy #work
  • Profile picture of the author OneManSEO
    My employer builds lots of WordPress websites and I've never found those development URLs indexed in Google when the Privacy setting is blocking search engines. Ever.

    Did you have a page that was coded outside of the WordPress CMS that was indexed instead? If you install Wordpress on say domain.com/blog instead of just domain.com, then I do not think your privacy settings will display on the root domain.
    {{ DiscussionBoard.errors[7465832].message }}
    • Profile picture of the author svalegria
      No. In fact it is only a three page site. Very thin (main, contact us, about) The moment it was created I put on the privacy before any content was added. The logs do not show a crawl during that minute or two. The description acknowledges that the robot.txt is on. This is the exact description under the title:
      " A description for this result is not available because of this site's robots.txt - learn more."

      Strange. I have used the privacy quite a bit while developing out a site. Never seen this before.
      {{ DiscussionBoard.errors[7465943].message }}
  • Profile picture of the author MikeFriedman
    Google, or any other bots for that matter, can ignore robots.txt. Best way to keep them out is .htaccess file and I don't think Wordpress changes the .htaccess file for that. I think it just uses robots.txt.

    Also, could be a case of the domain existing before maybe?
    {{ DiscussionBoard.errors[7466014].message }}
    • Profile picture of the author kaytav
      Originally Posted by MikeFriedman View Post

      Google, or any other bots for that matter, can ignore robots.txt. Best way to keep them out is .htaccess file and I don't think Wordpress changes the .htaccess file for that. I think it just uses robots.txt.

      Also, could be a case of the domain existing before maybe?
      I agree!! .htaccess file is considered as the master and all such changes should be first implemented here.
      {{ DiscussionBoard.errors[7469855].message }}
  • Profile picture of the author yukon
    Banned
    There's a small window of opportunity for Google to find the Index page from the instant you install WP & the time it takes you to login after the install to check that noindex box in the WP-admin.

    Google is very fast at finding new domains & fresh WP installs. Once Google has found the page & listed the page in the SERPs you'll have to wait for Google bot to return & see the noindex on the page/s. You can build a few quality external links to get Google to return to the page to see the noindex & then they'll remove your page/s from the SERPs.
    {{ DiscussionBoard.errors[7466408].message }}
  • Profile picture of the author petemcal
    Use as many signals as possible, htaccess, robots.txt and meta NOFOLLOW tags.

    If you're relying on a 3rd party make sure you check your coding to make sure it can't be indexed.
    Signature
    Follow Pete on Twitter #SEO #Marketing
    "It's like if Einstein did SEO"
    "Much shorter than Shakespeare"
    "I would follow Pete over Jesus Christ himself"
    {{ DiscussionBoard.errors[7470304].message }}
  • Profile picture of the author MatthewWoodward
    All wordpress does is add a sitewide noindex,nofollow tag and that has always worked for me without a problem!
    {{ DiscussionBoard.errors[7470949].message }}
  • Profile picture of the author Anurag96
    Than change your website's status from PRIVATE to PUBLIC.
    Signature
    Find Best Phones List, Tech Tips, Android Tricks and everything tech.
    Only on
    www.goingtechy.com
    {{ DiscussionBoard.errors[7471062].message }}
  • Profile picture of the author retsek
    The Wordpress Privacy setting works as it should. It adds a 'noindex' meta tag.

    From your post it looks like you're already blocking Google with robots.txt.

    noindex = allows crawling, but prevents indexing
    robots.txt = disallows crawling, but DOES NOT PREVENT INDEXING

    So using them in combination, makes Google UNABLE to see the noindex tag, because you have prevented them from crawling the page. If they know about the page from incoming links, they CAN index the page without having crawled it (or without being able to crawl it again if they already indexed it before).

    To fix the situation, remove the block in robots.txt and leave the noindex tag in place.
    {{ DiscussionBoard.errors[7471096].message }}

Trending Topics