robots.txt

9 years ago

you can take help of google create robots txt coding and use this your site on page you can set robots txt coding on which pages that you don't want crawl by google.

Thanks

{{ DiscussionBoard.errors[10075515].message }}

uditsh

9 years ago

Originally Posted by vithobaaseo

Hi friends I have small doubt regarding robots.txt. How to add robots.txt to our website? While analysing the website in woorank SEO tool its showing the website is missing robots.txt. Most of them would have done it previously please share me you ideas.. thanks

Robots.txt is a simple text file (.txt extension) that you need to upload at root of your website.
Search engines will look for a robots.txt file at the root of your website whenever they crawl your website.

Note: Please consult with a SEO specialist before uploading or making changes in it. Any improper or wrong robots.txt file may harm seo of your website.

Thanks

{{ DiscussionBoard.errors[10075590].message }}

MikeFriedman

9 years ago

Originally Posted by vithobaaseo

Hi friends I have small doubt regarding robots.txt. How to add robots.txt to our website? While analysing the website in woorank SEO tool its showing the website is missing robots.txt. Most of them would have done it previously please share me you ideas.. thanks

How do you offer SEO services (in your signature) and not know how to add a robots.txt file to a site?

WTF?

Thanks
1 reply

Signature

The SEO Pub Forums

For SEO news, discussions, tactics, and more.

{{ DiscussionBoard.errors[10075889].message }}

jamesfreddyc 9 years ago

Originally Posted by MikeFriedman

How do you offer SEO services (in your signature) and not know how to add a robots.txt file to a site?

WTF?

Sometimes I just peruse the SEO section of this place and it gives me comfort knowing I am competing with "SEO Experts".
- [ 1 ] Thanks
{{ DiscussionBoard.errors[10088249].message }}

kavyasharma561

9 years ago

Robots.txt is a file in the root directory of your web site that instructs web crawlers what parts, or all, or none of your site they are allowed examine.