6 {{ upvoteCount | shortNum }}
6 {{ upvoteCount | shortNum }}

What happens - If any website not have Robots.txt file ?

by Grettcs
5 replies
  • SEO
    • |
Hey warriors . i am here to ask regarding one query if one website having no robots.txt file and we are doing the seo on same . What are the disadvantages of not having robots.txt file in site and if we are doing the SEO without it on any website then, Do our seo work & efforts prove useful for the website partially or completely or not ?

Please put alight on my query with your relevant and informative answers.
#file #robotstxt #website
  • Profile picture of the author paulgl
    Knowledge is king.

    People will freak out though, if they see crawl errors.

    The googlebot will look first and foremost for a robots.txt,
    and give an unwarranted not found error. Makes some
    webmasters go ga-ga.

    Normal people, with normal websites, do not need a robots.txt.

    People think google is prone to show all sorts of stuff in
    SERPs, so they think they have to go crazy with it.
    Insane idea when you actually think about it.

    Paul
    Signature

    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[7664311].message }}
  • Profile picture of the author trevord92
    Also remember that only well behaved robots will obey robots.txt. If someone wants to scrape your site with their robot, they will happily disregard robots.txt

    Which leads onto the second point - don't use it to hide things like download links for your products as humans can read robots.txt files and get the links.
    Signature
    {{ DiscussionBoard.errors[7665181].message }}
    • Profile picture of the author paulgl
      Ya know, that's a good point. Google will readily index urls found
      on other sites, robots.txt or not.

      Any CMS like blogspot, WP, will set up a robots.txt to avoid
      indexing and wasting time on pages like search results. But
      for that to actually matter, you'd have to have a zillion people
      searching your blog at the same time the bot crawls it.

      Google has been in this search engine business for a long time.
      I can only presume they know what to do with relevant and
      irrelevant pages.

      If you don't want something found, then don't put it online,
      or behind a protected page.

      Paul
      Signature

      If you were disappointed in your results today, lower your standards tomorrow.

      {{ DiscussionBoard.errors[7665239].message }}
  • Profile picture of the author Grettcs
    Thanks all for getting me aware about this important issue.
    {{ DiscussionBoard.errors[7668545].message }}

Trending Topics