Go Back   WarriorForum - Internet Marketing Forums > The Warrior Forum > Adsense / PPC / SEO Discussion Forum
Register Blogs FAQ Social Groups CalendarHelp Desk

Reply
 
LinkBack Thread Tools
Old 12-05-2010, 03:56 PM   #1
HyperActive Warrior
War Room Member
 
Join Date: Oct 2009
Posts: 275
Thanks: 10
Thanked 4 Times in 4 Posts
Default How to link sitemap to the robots.txt?

Hi

This site is already indexed by google using both the .xml and the .xml.gz sitemaps. I thought I should update the robots.txt file correctly so that it also contain links to the sitemap. I just wonder if I should link the .xml version or the .xml.gz version

so basically which one of these should I use:
Sitemap: http://www.mydomain.com/sitemap.xml
Sitemap: http://www.mydomain.com/sitemap.xml.gz

I'm a bit worried as this site is 8 months old and I've trusted PC Robots.txt plugin in wordpress to do it's job, when I checked it displays an empty robots.txt so I uploaded a real robots.txt with the content I wanted. I've seen the error before but didn't pay much attention to it, but now as google don't displays ads on my site. I think it is somehow related to robots.txt issues...

any suggestions?
loginname is offline   Reply With Quote
Old 12-05-2010, 04:22 PM   #2
Plundering the Web
War Room Member
 
paulgl's Avatar
 
Join Date: Feb 2007
Location: , , .
Posts: 4,849
Thanks: 804
Thanked 1,199 Times in 886 Posts
Default Re: How to link sitemap to the robots.txt?

"How to link sitemap to the robots.txt?"

What for?

Completely useless. Every "major" (okay, like 2) automatically looks
for a robots.txt. If it's there, fine. If it's not there, fine.

You don't really need a robots.txt if you are like the 99.999% of the
rest of us.

You certainly don't need to link to it.

You don't really even need a sitemap, but that's sacrilegious to a lot
of SEO gurus.

There would only be an issue if your robots text told google to ignore
something. As stated, not really needed.

Make sure it's named robots.txt and in the root directory.

I see no reason to disallow anything. Google is a smart cookie.

If you put some garbage error stuff in it, my advice is to delete it,
or just leave it blank. I can't even see how an error in it would matter.
Search engine would ignore anything it can't figure out.

If you put:
User-agent: *
Disallow: /

That means it is not looking at squat. Get rid of the slash mark.

User-agent: *
Disallow:

That means look at everything. But what's the point?

Best advice: Don't have a robots.txt
Done and done.

If that does not fix the google ad thing, you have a problem somewhere else.

But I can't fathom robots.txt causing google to not show ads anyway.
That may be another matter.

Paul

How to Make Money off Facebook: Login to your account. Deactivate your account. Get your butt to work.
paulgl is offline   Reply With Quote
Old 12-05-2010, 05:12 PM   #3
HyperActive Warrior
 
Join Date: Nov 2010
Posts: 343
Thanks: 20
Thanked 20 Times in 20 Posts
Default Re: How to link sitemap to the robots.txt?

In your robots.txt file, add this line :
Sitemap: http://www.mydomain.com/sitemap.xml

The gunzipped version is smaller to download, so you can add that too. But I have never tried GZ version. Why dont you add the sitemap in Google Webmaster Tools and check if it can retrieve and extract the URLs.

gtk29 is offline   Reply With Quote
Old 12-05-2010, 05:28 PM   #4
SEO Strategist
War Room Member
 
yukon's Avatar
 
Join Date: Jun 2010
Posts: 6,532
Thanks: 355
Thanked 1,992 Times in 1,273 Posts
Default Re: How to link sitemap to the robots.txt?

I run:

Sitemap: http://domain.com/sitemap.xml.gz

Works just fine.
yukon is offline   Reply With Quote
Old 12-07-2010, 10:29 PM   #5
Warrior Member
 
Join Date: Dec 2010
Posts: 4
Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: How to link sitemap to the robots.txt?

Site link is not in itself a symbol of the REP. Which is only there to help our crawlers find your map. usual rules apply EPR, then you can block robots from indexing your site.

bhanjerikley is offline   Reply With Quote
Reply

  WarriorForum - Internet Marketing Forums > The Warrior Forum > Adsense / PPC / SEO Discussion Forum

Tags
link, robotstxt, sitemap

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off



All times are GMT -6. The time now is 10:35 PM.