Robots.txt???

by Ruby Tyagi

Posted: 14 years ago 17 replies

what is Robots.txt??? how it is used???

#robotstxt

outdoorfountains 14 years ago

robot.txt is a file that allow and disallow your website pages from search engines crawling.
- Thanks
Signature
outdoor fountains- Outdoor fountain
{{ DiscussionBoard.errors[5434668].message }}
Pawpoint 14 years ago

You can use a special robots.txt to tell search engine spiders not to index the content of a page, and/or not scan it for links to follow. You dont have to use it, but some people like parts of their pages ignored by the spiders if they are not relevant to what the site is about
- Thanks
Signature
http://www.cheapherbalincense.biz Cheap UK Strong Aroma Incense + bulk - Free UK delivery 1.5g to 1Kg
{{ DiscussionBoard.errors[5434680].message }}
mukeshkumar@ 14 years ago

Robots.txt file is used to request Search Engine Spiders about the pages you want them to crawl or index or not.
It is an important factor for the webpages to be indexed properly.
- Thanks
Signature
Your trusted partner for Digital Media Solutions | Offshore Digital Marketing Company
{{ DiscussionBoard.errors[5434704].message }}
ghazia 14 years ago

Robots.txt is a text file which can be uploaded in the root folder of your site. Sometimes you don't want the robots just roaming anywhere they like on your site.You can use robots.txt to block some pages on your site not be crawled by crawlers. Have a visit at this link to know about the effective use of robots.txt How to Write a robots.txt File
- Thanks
{{ DiscussionBoard.errors[5434784].message }}
olalinks Banned 14 years ago

[DELETED]

{{ DiscussionBoard.errors[5434803].message }}
- Braylen 14 years ago
  
  Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not.
  
  Thanks
  
  {{ DiscussionBoard.errors[5434898].message }}
ricky pounting 14 years ago

rebotex is file use to restrict google crawler to index your website pages
- Thanks
{{ DiscussionBoard.errors[5434934].message }}
John Conner 14 years ago

Robots.txt is a text files that used to tells engines that what to index and what not to index on a certain websites.

It is written as:

User-agent: *
Disallow: /
- Thanks
Signature
TranscriptionServicesIndia.Com (TSI) - Low cost, fast and accurate transcription services for interviews, podcasts, webinars, dictations, etc.
DataExtractionServices.Com - Scraping data from web directories, WebPages, LinkedIn, Yelp, Yell, Amazon, eBay etc.
{{ DiscussionBoard.errors[5434941].message }}
Webster Logan 14 years ago

A site that hides part that user doesn't want to be crawled by Google.
- Thanks
{{ DiscussionBoard.errors[5434970].message }}
webpageworker 14 years ago

hi
robot.txt is a file that allow and disallow your website pages from search engines crawling. So if ur page is in process disallow it..
- Thanks
{{ DiscussionBoard.errors[5436582].message }}
Ruby Tyagi 14 years ago

Thanx for giving info..but how i will create it??
- Thanks
- 1 reply
{{ DiscussionBoard.errors[5440906].message }}
- John Conner 14 years ago
  
  Originally Posted by Ruby Tyagi
  
  Thanx for giving info..but how i will create it??
  
  Hey Ruby,
  
  I already answered your question in above my thread.
  
  Thanks
  
  Signature
  TranscriptionServicesIndia.Com (TSI) - Low cost, fast and accurate transcription services for interviews, podcasts, webinars, dictations, etc.
  DataExtractionServices.Com - Scraping data from web directories, WebPages, LinkedIn, Yelp, Yell, Amazon, eBay etc.
  
  {{ DiscussionBoard.errors[5442603].message }}
Yulia from DNP 14 years ago

I think you need to read very detailed manual : The Web Robots Pages
- Thanks
Signature

Yulia borova
Affiliate Manager | CPA Affiliates Network.
$50 Signup Bonus â€“ Faster approval for Warrior forum members
Email: Yulia@DirectNetPartners.com
{{ DiscussionBoard.errors[5441874].message }}
seanpearse 14 years ago

here's a handy robots.txt generator.....Robots.txt Generator - McAnerin International Inc.
- Thanks
Signature

What the world needs is more geniuses with humility, there are so few of us left!

Local SEO Ireland
{{ DiscussionBoard.errors[5444103].message }}
Ruby Tyagi 14 years ago

Thanx all of you in making me understand one new thing..
- Thanks
{{ DiscussionBoard.errors[5448356].message }}
mogulskiholiday 14 years ago

A robots.txt is a permissions file that can be used to control which webpages of a website a search engine indexes. The file must be located in the root directory of the website for a search engine website-indexing program (spider) to reference,
- Thanks
- 1 reply
{{ DiscussionBoard.errors[5448601].message }}
- AlbertSmiths 14 years ago
  
  The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this...
  
  /robots.txt
  
  Thanks
  
  1 reply
  
  {{ DiscussionBoard.errors[5450067].message }}
  
  Webster Logan 14 years ago
  
  What is the kind of information that site owners does not want to be crawled?
  
  Thanks
  
  {{ DiscussionBoard.errors[5452822].message }}

Robots.txt???

Trending Topics

The Winter Olympics....

I need tips about casino affiliate marketing

I have an app - now what?

What is the future of SEO and Blog?

Page B ranking for my target keyword instead of Page A