9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

Guarantee No Crawlers Access My Site

by packerfan
8 replies
I need to put a copy of my site on either a second domain or a sub domain (like some opinion here) because I'm going to be testing a conversion to a responsive design of the site.

I'd also like to have a "playground" so I can test without worry of google crawling the site.

I know in wordpress I can tell it to not allow search engines, and I can use a robot.txt file, but I've heard both aren't guarantees.

Is there some way I can be sure the site won't get crawled?

Also - is it better to use a subdomain or a completely separate domain for this? I have a few domains not in use so that's not an issue if it's better.

Thanks,
Paul
#access #crawlers #guarantee #site
  • Profile picture of the author mojojuju
    If you're going to be the only one accessing the site, you could only allow access to your IP.

    Another thing you can do is create a domain on your server with an extension that isn't real, like example.dev or example.test. Then in your hosts file on your computer, just add that fake domain and the corresponding IP address of your site. Only you will be able to go to it.
    Signature

    :)

    {{ DiscussionBoard.errors[8807213].message }}
    • Profile picture of the author packerfan
      How do i go about limiting access to the IPs that I want? There's a chance I will hire a developer and would need to grant them access as well.
      Signature

      Nothing to see here

      {{ DiscussionBoard.errors[8807247].message }}
      • Profile picture of the author mojojuju
        Originally Posted by packerfan View Post

        How do i go about limiting access to the IPs that I want? There's a chance I will hire a developer and would need to grant them access as well.
        You can put the following in an .htaccess file if your hosting supports it:


        Order Deny,Allow
        Deny from all
        Allow from 1.2.3.4,1.2.3.5

        1.2.3.4 and 1.2.3.5 would be the IP addresses of people you want to allow. Everyone else will be blocked.

        Additionally, I think cPanel hosts allow you to block or allow IPs directly from cPanel.
        Signature

        :)

        {{ DiscussionBoard.errors[8807273].message }}
        • Profile picture of the author packerfan
          Originally Posted by mojojuju View Post

          You can put the following in an .htaccess file if your hosting supports it:


          Order Deny,Allow
          Deny from all
          Allow from 1.2.3.4,1.2.3.5

          1.2.3.4 and 1.2.3.5 would be the IP addresses of people you want to allow. Everyone else will be blocked.

          Additionally, I think cPanel hosts allow you to block or allow IPs directly from cPanel.
          Perfect, I have a cPanel host and will look into it. Thanks a lot for your help! Now to go figure out how to make my thesis 1.8.2 implementation of Thesis responsive. Should be fun.
          Signature

          Nothing to see here

          {{ DiscussionBoard.errors[8807307].message }}
  • Profile picture of the author David Beroff
    Easiest thing to do here, IMHO, is to use simple http authentication. Just put a password at the top of the site (subdomain or separate domain; doesn't matter) and no one will be able to see any of it. I do it all the time. It should be supported by cPanel, or just do it manually; it only involves two files.
    Signature
    Put MY voice on YOUR video: AwesomeAmericanAudio.com
    {{ DiscussionBoard.errors[8807469].message }}
    • Profile picture of the author packerfan
      Originally Posted by David Beroff View Post

      Easiest thing to do here, IMHO, is to use simple http authentication. Just put a password at the top of the site (subdomain or separate domain; doesn't matter) and no one will be able to see any of it. I do it all the time. It should be supported by cPanel, or just do it manually; it only involves two files.
      Can you provide an example of those files?
      Signature

      Nothing to see here

      {{ DiscussionBoard.errors[8807615].message }}
      • Profile picture of the author David Beroff
        Originally Posted by packerfan View Post

        Can you provide an example of those files?
        These will be slightly different for each host, but here's how I do it on Rackspace. Your host may have a similar set of instructions. Don't forget the leading dots on the filenames; those are crucial.
        Signature
        Put MY voice on YOUR video: AwesomeAmericanAudio.com
        {{ DiscussionBoard.errors[8807662].message }}
  • Profile picture of the author nettiapina
    I'm using the trick @mojojuju wrote about. If you want to have a website that's on a server but that no-one else can easily access, just use the hosts file on your computer to override local name server settings.

    For site building I'm often using real subdomain addresses so that I can add them to the public DNS when it's the time to show the result to the client. Something like client.mycompany.tld.
    Signature
    Links in signature will not help your SEO. Not on this site, and not on any other forum.
    Who told me this? An ex Google web spam engineer.

    What's your excuse?
    {{ DiscussionBoard.errors[8808910].message }}

Trending Topics