Please (Pretty Please) Help With Robots.txt File On Site--I've Tried Everything

by TMW
3 replies
Hello everyone. I would be eternally grateful for some ideas on this perplexing problem. I am not a programmer and know enough code to write a link and a few other things and that's about it. But I have worked with WordPress quite a bit so in general don't usually run into problems.

So this particular site is airconditioning-orlando.net. My husband and I own this small business and I have worked my butt off trying to get his phone to ring! We've had moderate success on an older site but this new site is killing my efforts at the moment.

The deal is that if you run a Google search like this:

info:airconditioning-orlando.net

The results say that Google cannot index bc of a robots.txt file (and site has not appeared anywhere on Google that I have seen). But there is no robots.txt file!!! I have Googled this issue and also called my host. Other people have sometimes had this problem with Google and I've tried one suggested fix, which was a plug-in which allowed me to add info to supposedly allow/disallow, but I still don't think it added a robots.txt file. I also had the host add a blank robots.txt file, which I could then edit in Yoast. But, still, no luck.

This is driving me batty. I have reached the point where I'm thinking about switching this to an entirely new domain but even with importing this site would take a LOT of tweaking to get back to where it is now. I have found a mirroring plug-in (mirrors everything with the push of one button including theme customizations, plug-ins and internal links). My concern about the mirroring plug in is that it might mirror everything exactly--including whatever is causing the problem. Urrrggghhhh!

One last thing is that I just installed the plug-in that allowed me to add the allow coding to the settings. This was more than 24 hours ago. I read somewhere that Google would update and fix this problem (if it was going to be fixed) within twenty four hours. So that hasn't happened. Have I maybe not waited long enough?

FYI the reason this might have started is because I initially checked "discourage search engines from indexing this site," then I unchecked it, then I checked it again while building. Maybe the back and forth screwed something up. Can anyone help? Should I just move to a new domain? I have an equally decent name (not that this one is all that great anyway) and this is a new site so moving not a biggie except I don't want to import same problem.

Thank you for reading this mess and thank you even more for any and all suggestions or ideas.
#file #pretty #robotstxt #siteive
  • Profile picture of the author David V
    Actually you have over 8 pages in google. (not listings, but google pages, pagination).
    Search like this for listed pages....
    site:airconditioning-orlando.net

    You do have a robots.txt in the root.

    This is it:
    Code:
    User-Agent: *
    Disallow: 
    Allow: /airconditioning-orlando.net
    I would recommend you read about the robots.txt file a little bit.
    Robotstxt.org

    That "Allow" line you have is not valid anyway. You don't have to allow, it's the default.
    Just disallow what you don't want crawled.
    The link above will show you many examples to do that.
    You'll need to FTP into your server root to change it, or use a plugin.

    Edit:
    Here's a basic starter that blocks the basics in WordPress that should not be listed anyway.
    This will not block any pages, just the "wp" folders (admin, etc..), and files with the .php extension, the feed, the trackback, and the cgi-bin.
    You may need to add more depending on what you have for directories that should be blocked.


    Code:
    User-agent: *
    Disallow: /wp-*
    Disallow: *.php
    Disallow: */feed*
    Disallow: */trackback*
    Disallow: /cgi-bin/
    {{ DiscussionBoard.errors[7832161].message }}
  • Profile picture of the author SmallBizWebsites
    I am curious. Why do you think you need a robots.txt file at all? Nearly all sites have no need whatsoever for this file, and you may be better off deleting it completely unless there is a solid reason for having it.

    If you will permit me to offer a suggestion or two, the text on your Home page is very hard to read. You have over-used bold text. Also, your paragraph headings look odd- they are light grey on a white background, which makes for an odd style.

    If I may also suggest, the page titles and meta descriptions are spammy and go against Google's guidelines. Here is a sample page title:

    Product Updates Archives | Air Conditioning OrlandoAir Conditioning Orlando

    You will find a link to Google's guidelines here:
    Site title and description - Webmaster Tools Help
    {{ DiscussionBoard.errors[7843062].message }}
    • Profile picture of the author Greg71
      Slow down a bit. Uncheck the block search engines thing and leave it forever, even if you rebuild/ edit.

      Google takes a few days to do anything so just be patient and wait for your changes to appear.

      Your Robots.txt file should read like this:

      User-agent: *
      Disallow: /wp-admin/
      Disallow: /wp-includes/

      Sitemap: http://www.yoursite.com/sitemap.xml.gz (or whatever your sitemap is called)

      The sitemap is optional, not necessary. The disallows here prevent G from bothering with Wordpress admin directories. Again, optional.

      This is how one of my working-normally sites is configured.
      {{ DiscussionBoard.errors[7852612].message }}

Trending Topics