11 {{ upvoteCount | shortNum }}
11 {{ upvoteCount | shortNum }}

Some Robot.txt Questions

by SocialDemon
9 replies
  • SEO
    • |
I want to change my robot.txt file to optimize my website. I spent hours to find in Google my desired info. Unfortunately I did not get enough good info. So asking question here assuming this is the best place to know about onpage optimization and robot.txt customization. My questions are, how can I change robot.txt file to make ....

All Internal links - Dofollow
All outgoing links - Nofollow
Block Moz, Ahref, SEO Spyglass, Majestic SEO robots
#questions #robottxt
  • Profile picture of the author UMS
    Short answer, is that you can't do any of that with a robots.txt file. That's not what it's for.

    It simple tells search engines which parts of a website they should or should not index. Note that it is simply an advisory and some search engines and many indexing software simply ignore it.
    Signature
    {{ DiscussionBoard.errors[9177364].message }}
    • Profile picture of the author SocialDemon
      Originally Posted by UMS View Post

      So answer, is that you can't do any of that with a robots.txt file. That's not what it's for.

      It simple tells search engines which parts of a website they should or should not index. Note that it is simply an advisory and some search engines and many indexing software simply ignore it.

      Yes I know. I can request them but it all upto the robots. If any robot wants to ignore my request then I have nothing to do. But from my place I will do my job.

      Can you please tell me what to write on my robot.txt file to nofollow all outgoing links and dofollow all internal links?
      {{ DiscussionBoard.errors[9177401].message }}
  • Profile picture of the author Daones
    You can block specific bots as you mentioned like ahrefs as long as you know the bots name im not sure of all of the bots names that you mentioned but if you google them you might be able to find their name.

    For ahrefs they already have an article to block their bot:
    https://ahrefs.com/robot/

    So i know you can block ahrefs as they are more commercial, some lesser known services you might not be able to block.

    For internal links and outgoing links you have to deal with them on per case basis. Internal links easy... just keep it as it is most like are dofollow by default.

    For outgoing links depending on the platform you can get a nofollow plugin on wordpress or simply add the rel=nofollow tag on outgoing links. If you have a ton of outgoing links on your site and many pages someone with web skills can probably give feedback on a script to change all of them to nofollow and ignore links containing your own domain name.
    Signature

    Diverse Link Building Package Get Improved Rankings That Stick - 20% Cheaper Prices to WF Members
    {{ DiscussionBoard.errors[9177369].message }}
  • Profile picture of the author SocialDemon
    Anyone knows how to Nofollow all outgoing links and Dofollow all internal links in robot.txt file?
    {{ DiscussionBoard.errors[9177793].message }}
  • Profile picture of the author paulgl
    Did you know that the best way to optimize a robots.txt is to
    either not have one, or leave it blank?

    Big secret, I suppose.

    Paul
    Signature

    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[9179593].message }}
  • Profile picture of the author Earl Gray
    Robots file just says about what links, or url structure pattern exclude or include from index, and declare path to your sitemap.
    If you want to make all your link nofollow at once, there are JS solutions. I suppose they work for Google, but still, since it's JS and I didn't try it personally, I can't say for sure.

    Or if isn't a huge project, you might search through templates, find string "a href" and replace it with "a rel='nofollow' a href=". Ir if you are using URL generator function, change it..
    {{ DiscussionBoard.errors[9183599].message }}
  • Profile picture of the author thebert
    There is no "big secret" to the robots.txt file. None, nada. zilch.

    1. You cannot alter the nature of your incoming and outgoing links (ie: dofollow, nofollow) using a robots.txt file.
    2. You can limit access to none, some or all of your site using a robots.txt file. You need to know how to use regular expressions (regex) and you need to know how to script regex within the robots.txt file.

    Here's a simple guide:
    A guide to robots.txt files
    Here's a slighly more complicated guide:
    The Web Robots Pages
    If you really get excited, here's a link to bot signatures that you can use:
    The Web Robots Pages

    Lastly, I use the robots.txt file to control who, or what, can reverse-engineer my back-linking strategies. PM me if want to know more about that!

    Good luck!
    {{ DiscussionBoard.errors[9183693].message }}
    • Profile picture of the author Earl Gray
      Originally Posted by thebert View Post

      You need to know how to use regular expressions (regex) and you need to know how to script regex within the robots.txt file.
      As far as I know, you can't use regex in robots.txt. There is a simple logic, but it isn't regex.

      And I would have one question, regarding changing all outgoing links to nofollow. If anyone did this, i would like to know your results to SERP.
      I red 2 years ago on a some guy's blog who said, that his site had a bit lower visits since he changed all links to nofollow, major of links were comments.
      I'm doing something similar on my websites, and just wondering whats your experience.
      {{ DiscussionBoard.errors[9183828].message }}

Trending Topics