Need help in customizing robots.txt File

14 years ago

Here's an example of a proper robots.txt

http://www.hovercoupon.com/robots.txt

Hope that helps. You only Disallow stuff you want disallowed... IF you Disallow / - Google ignores the whole site.

Thanks

{{ DiscussionBoard.errors[2471488].message }}

rajanpanicker

14 years ago

So should I disallow Sitemap
MY site shows like above where my Sitemap is shown along with the Disallow tab

Thanks

{{ DiscussionBoard.errors[2474574].message }}

HunterSnake

14 years ago

No... you don't want to disallow your sitemap. That kind of defeats the purpose of creating a sitemap.

Thanks
1 reply

{{ DiscussionBoard.errors[2474632].message }}

rajanpanicker

14 years ago

Originally Posted by HunterSnake

No... you don't want to disallow your sitemap. That kind of defeats the purpose of creating a sitemap.

So what exactly does this mean.My robots.txt file shows

User-agent: *
Disallow:

Sitemap: http://www.urladdress.net/sitemap.xml

Now does that mean that my robots.txt is set to Disallow my Sitemap,so should I remove the Sitemap: http://www.urladdress.net/sitemap.xml

Thanks
1 reply

{{ DiscussionBoard.errors[2479648].message }}

D Baker

14 years ago

No,

This is telling the robots where your sitemap is located so they can find it faster.

Read this post:

Must See SEO Guide for All WordPress Bloggers

It can help you better understand the robots.txt file among other things. It also give you an exact example of how to structure your robots.txt file.

You can also simply install the PC Robots.txt Wordpress plugin - get it here: WordPress › PC Robots.txt WordPress Plugins

Thanks
1 reply

Signature

SENuke Review
SENuke XCr Review
Article Builder
Unique Article Wizard

{{ DiscussionBoard.errors[2480351].message }}

rajanpanicker

14 years ago

Originally Posted by D Baker

No,

This is telling the robots where your sitemap is located so they can find it faster.

Read this post:

Must See SEO Guide for All WordPress Bloggers

It can help you better understand the robots.txt file among other things. It also give you an exact example of how to structure your robots.txt file.

You can also simply install the PC Robots.txt Wordpress plugin - get it here: WordPress › PC Robots.txt WordPress Plugins

Ok so
Disallow:
means its not Disallowing any of my Content right.So I dont need to do anything with my robots.txt right and Sitemap Content is showed in a different line?

Thanks

{{ DiscussionBoard.errors[2483874].message }}

mozzer

14 years ago

I recommend the iRobots.txt SEO Wordpress plugin.

Thanks

{{ DiscussionBoard.errors[2486154].message }}

rajanpanicker

14 years ago

No,I just want to know what exactly is my robots.txt saying.Do I need to edit any text or should I just leave it like that.And I believe the Site map line is in the next line after the Disallow line right?

Thanks

{{ DiscussionBoard.errors[2493323].message }}

Need help in customizing robots.txt File

Trending Topics

Any thoughts on golf's latest phenom...Scottie Scheffler??

The Twilight Zone

Should i Index Author pages?

best high quality crypto traffic.

What's the Best IM-Related Skill to Learn Today That Can Help in the Future?