9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

Anoying Robots.txt

by jcdmp
8 replies
Hi my friends

I have a few blogs out and like many ppl im getting a huge drop on serps for a lot of them.

I used the same Robots.txt for a lot of time but i heard that a empty one would be the best choice but i dont want a few things on my blog indexed.

I started making some tests with an empty Robots.txt and didnt notice any changes and last night someone just sent me a Robots.txt that he say its has the best options...and today i got a lot of "warnings" around 20...in my Google Webmaster Tools....

So i dont know what to do...empty?...full blocking a lot of things?....

I need help on this one...dont know what to choose

If ppl have a Robots.txt that use without any problems i would like to test it...or just point me the right direction


Thanks a lot
#anoying #robotstxt
  • Profile picture of the author John Romaine
    Im by no means an expert on this subject, but for what I do, it works well. I typically just keep it simple.

    Heres an example...

    xxxx indicates directories you want to resctrict
    xxx.html indicate pages you want to restrict also

    At the very bottom you should specify the full path of your XML sitemap.

    Code:
    User-Agent: *
 
Disallow: /xxxxx/
Disallow: /xxxxx/
Disallow: /xxxx.html
 
Sitemap: http://www.yoursite.com/sitemap.xml
    More information available here...

    http://www.robotstxt.org/
    Signature

    BS free SEO services, training and advice - SEO Point

    {{ DiscussionBoard.errors[6000770].message }}
    • Profile picture of the author Ralf Skirr
      The robots.txt is of little importance to SE rankings - unless you get it wrong.

      Do NOT publish robots.txt with contents you don't understand, because someone told you it's good. If it's wrong, you can get your stuff deindexed.

      The warnings on your webmaster account may or may not be important. That is hard to say. They also give you warnings about the things you want to be blocked.
      {{ DiscussionBoard.errors[6001057].message }}
      • Profile picture of the author John Romaine
        Something tells me he's probably "disallowed" everything!
        Signature

        BS free SEO services, training and advice - SEO Point

        {{ DiscussionBoard.errors[6001065].message }}
  • Profile picture of the author Michael Meaney
    Webmaster tools will generate a robots.txt file for you, if you copy & paste your existing one here myself and others will be able to tell you if it's right or not.

    But the robots.txt might not be the reason your sites have dropped, Google recently updated their algorithm.
    Signature
    {{ DiscussionBoard.errors[6001089].message }}
  • Profile picture of the author rooze
    The file is to provide instructions to those search engines which read it, why would you have an empty file?
    Taking a Robots file from another website is a recipe for disaster. Does the other site have the exact same file and directory structure as your site? I doubt it. No wonder Google WT is having a melt down.

    If you're not sure what you're doing, just delete the file. Unless you have something very specific which you DO NOT want to be indexed, there's no real need for it anyway.
    Signature
    {{ DiscussionBoard.errors[6001093].message }}
  • Profile picture of the author jcdmp
    thanks to all the ppl that replied and helped me to understand this a little more

    this what im using...very simple i think

    User-agent: Googlebot
    Disallow: /wp-content/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /archives/
    Disallow: /*?
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: */trackback/

    User-agent: Googlebot-Image
    Disallow: /wp-includes/

    User-agent: Mediapartners-Google*
    Disallow:

    User-agent: ia_archiver
    Disallow: /

    User-agent: duggmirror
    Disallow: /

    thanks
    {{ DiscussionBoard.errors[6002102].message }}
  • Profile picture of the author Baadier Sydow
    I agree with Rooze, use KISS metaphor, Keep It Simple Stupid.
    Signature
    Baadier Sydow
    Budget Furniture

    {{ DiscussionBoard.errors[6002245].message }}
  • Profile picture of the author Michael Meaney
    Try this:

    User-agent: *
    Disallow:
    Sitemap: http://www.YOURSITE.com/sitemap.gz
    This allows everything, everywhere. The sitemap is optional.
    Signature
    {{ DiscussionBoard.errors[6002688].message }}

Trending Topics