7 {{ upvoteCount | shortNum }}
7 {{ upvoteCount | shortNum }}

Robots.txt

by abhibangal
6 replies
  • SEO
    • |
I was just browsing thorugh my Google WM tools and I found through robots.txt tester that my robots file is empty. I compared it with another site and saw that there was the (xml) sitemap address.
Do I need to have something similar to that? Please help me. I am sure this will have a difference on SEO and some traffic due to crawling.
#robotstxt
  • Profile picture of the author Michael Nguyen
    Dont just copy people. Tell us why you think you need to edit robots.txt.

    If you want to tell the bots to not crawl a folder/directory then you can say so in the txt file. I suggest you Google what a robots.txt file first before you decide if you need it.
    Signature
    FREE Cold Calling Course: >>> Looking for Serious & Aspiring Cold Callers to Train for FREE <<<
    {{ DiscussionBoard.errors[9562145].message }}
    • Profile picture of the author abhibangal
      Originally Posted by Michael Nguyen View Post

      Dont just copy people. Tell us why you think you need to edit robots.txt.

      If you want to tell the bots to not crawl a folder/directory then you can say so in the txt file. I suggest you Google what a robots.txt file first before you decide if you need it.
      Isn't the robots file not needed (I assume we all need it for SEO)? Secondly, it is not the question of copying. My txt file shows 0 bytes. So, it's equal to non-existing, ins't it?
      {{ DiscussionBoard.errors[9562373].message }}
      • Profile picture of the author Khyati
        If you check google support in your webmaster tools you will see in detail why and how to use robots.txt
        {{ DiscussionBoard.errors[9562701].message }}
  • Profile picture of the author SEO Power
    What did you build your website with? If it was built with WordPress, it wouldn't be empty as all WordPress installations contain robots.txt files with some commands.

    If your website is a pure html-based site, it's normal for the robots.txt file to be empty. Besides, an empty robots.txt won't hurt your site SEO-wise. You can leave it empty if you want to avoid 404 errors and, at the same time, don't want to give bots any specific commands.

    The XML sitemap address is also not necessary unless your site has a valid XML sitemap - and an XML sitemap is not necessary if you have a properly organized site.
    Signature
    ONE HOUR INDEXING - STOP WASTING MONEY ON LINKS GOOGLE NEVER FINDS
    {{ DiscussionBoard.errors[9562995].message }}
  • Profile picture of the author FreedomBlogger
    I recommend to use a robots.txt file to be more search engine friendly but it is not very necessary to get free organic traffic from the search engines.

    the reply above ^^ is right on! .. you can always go to the Google support to learn more on this matter. They have all the answer, and after all, this is something from Google right!

    I hope this helps!

    I wish you the best!

    Cheers!
    Signature
    At the beginning, I thought making money online with a blog was super super hard. Not anymore. Learn the art of making money online blogging - step by step - HERE.
    {{ DiscussionBoard.errors[9563173].message }}
  • Profile picture of the author aadi14
    Originally Posted by abhibangal View Post

    I was just browsing thorugh my Google WM tools and I found through robots.txt tester that my robots file is empty. I compared it with another site and saw that there was the (xml) sitemap address.
    Do I need to have something similar to that? Please help me. I am sure this will have a difference on SEO and some traffic due to crawling.
    People generally leave the robots.txt field blank. They only disallow something that is meant to be blocked for example - a shopping card page, payment details etc.

    If you don't even have a robots file you can still do well.

    And placing an xml sitemap will have no effect either. So just check first whether there are pages that you want to block or not. If not let the file be empty.
    Signature
    Ecommerce Website Designing Company | Bespoke Website & Application Development Company
    {{ DiscussionBoard.errors[9564059].message }}

Trending Topics