16 {{ upvoteCount | shortNum }}
16 {{ upvoteCount | shortNum }}

What exactly is Robot.txt?

by brian culbert
14 replies
  • WEB DESIGN
    • |
After I publish a WP website on my own host server, I discover at logs of IIS, that there is a file called "robots.txt", some IPs try to check it on my website.
I searched for that and I knew that is for search engine spiders.
What the best thing I shall do to get all my web site indexed?
If I allow all spiders to see all files, is that harm my website anyway?
#robot.txt #robottxt
  • Profile picture of the author mike-seo
    Not spiders will harm your website, but hackers will do
    {{ DiscussionBoard.errors[3206524].message }}
  • Profile picture of the author ckmcharles
    Originally Posted by brian culbert View Post

    What the best thing I shall do to get all my web site indexed?
    Search for "website indexer" on Google. The software can help your site get indexed in less than 48 hours (most cases) and it's free.

    Originally Posted by brian culbert View Post

    If I allow all spiders to see all files, is that harm my website anyway?
    It will "harm your website" if there is any file or page you want to be hidden from public i.e. a product you're selling or giving away to subscribers...
    {{ DiscussionBoard.errors[3206903].message }}
  • Profile picture of the author bhijackwilliams
    Robots.txt is a file that allows you to control the search engines to scan or not to scan certain parts of your site. Google in particular, follow the instructions in the robots.txt file.
    {{ DiscussionBoard.errors[3208973].message }}
  • Profile picture of the author onlineprime08
    Robots.txt is a file that allows you to control the search engines to scan or not to scan certain parts of your site. Google in particular, follow the instructions in the robots.txt file.
    You mean "follow" or "no follow" isn't it?
    {{ DiscussionBoard.errors[3209525].message }}
  • Profile picture of the author rogervickers
    robots.txt file is a file to guide the search engine crawlers to crawl our site pages.
    {{ DiscussionBoard.errors[3210002].message }}
  • Profile picture of the author keyideas
    Robots.txt file gives permissions to search engine for which folder or files have to crawl or not.
    {{ DiscussionBoard.errors[3210079].message }}
    • Profile picture of the author jalicia18
      Robots.txt is a file to control which folder or file to get indexed in the Search Engine or Spiders. You can tell the spiders or User-Agent if you want the file or folder to show up on the search results.
      Example, if you don't like the "image" folder to be scanned by spiders or search engine crawlers:

      User-agent: *
      Disallow: /image
      Signature

      For your call center needs:
      Call Center Philippines
      BPO Services Philippines
      Philippines Call Center

      {{ DiscussionBoard.errors[3216179].message }}
  • Profile picture of the author xrvel
    In short, there is no harm on it
    In long, if exists, it can manage how SE is allowed / disallowed to visit some part of your website.
    {{ DiscussionBoard.errors[3216195].message }}
  • Profile picture of the author nsdoi
    Hello.........

    Robot.txt is simple text file which is used to tell search engine that which content you would not like to visit them on you website..........
    {{ DiscussionBoard.errors[3221947].message }}
  • Profile picture of the author alcymart
    Read here for the answer: The Web Robots Pages

    Take care,

    Bernard St-Pierre
    Marketing Consultant
    Copywriter/Teacher
    Signature
    {{ DiscussionBoard.errors[3221964].message }}
  • Profile picture of the author chimeswebdesign
    Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
    {{ DiscussionBoard.errors[3297238].message }}
  • Profile picture of the author carry12
    Hello Friends........

    "Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all

    Thanks
    Signature
    GreatForum
    {{ DiscussionBoard.errors[3297683].message }}
  • Profile picture of the author lisaroy1
    It is also known as Robots Exclusion Protocol or robots.txt protocol,It is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable....In simple words This is a file written and stored in the root directory of a website that restricts search engine spiders from indexing certain areas of the website.
    {{ DiscussionBoard.errors[3297774].message }}
    • Profile picture of the author bradlean
      I have a blogger.com blog and I submitted to google webmaster but I want to change the permission about robot.txt...

      How? or do I have to have my own webhosting not a blogger?
      {{ DiscussionBoard.errors[3325676].message }}

Trending Topics