How to link sitemap to the robots.txt?

4 replies
  • SEO
  • |
Hi

This site is already indexed by google using both the .xml and the .xml.gz sitemaps. I thought I should update the robots.txt file correctly so that it also contain links to the sitemap. I just wonder if I should link the .xml version or the .xml.gz version

so basically which one of these should I use:
Sitemap: http://www.mydomain.com/sitemap.xml
Sitemap: http://www.mydomain.com/sitemap.xml.gz

I'm a bit worried as this site is 8 months old and I've trusted PC Robots.txt plugin in wordpress to do it's job, when I checked it displays an empty robots.txt so I uploaded a real robots.txt with the content I wanted. I've seen the error before but didn't pay much attention to it, but now as google don't displays ads on my site. I think it is somehow related to robots.txt issues...

any suggestions?
#link #robotstxt #sitemap
  • Profile picture of the author paulgl
    "How to link sitemap to the robots.txt?"

    What for?

    Completely useless. Every "major" (okay, like 2) automatically looks
    for a robots.txt. If it's there, fine. If it's not there, fine.

    You don't really need a robots.txt if you are like the 99.999% of the
    rest of us.

    You certainly don't need to link to it.

    You don't really even need a sitemap, but that's sacrilegious to a lot
    of SEO gurus.

    There would only be an issue if your robots text told google to ignore
    something. As stated, not really needed.

    Make sure it's named robots.txt and in the root directory.

    I see no reason to disallow anything. Google is a smart cookie.

    If you put some garbage error stuff in it, my advice is to delete it,
    or just leave it blank. I can't even see how an error in it would matter.
    Search engine would ignore anything it can't figure out.

    If you put:
    User-agent: *
    Disallow: /

    That means it is not looking at squat. Get rid of the slash mark.

    User-agent: *
    Disallow:

    That means look at everything. But what's the point?

    Best advice: Don't have a robots.txt
    Done and done.

    If that does not fix the google ad thing, you have a problem somewhere else.

    But I can't fathom robots.txt causing google to not show ads anyway.
    That may be another matter.

    Paul
    Signature

    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[2969117].message }}
  • Profile picture of the author gtk29
    In your robots.txt file, add this line :
    Sitemap: http://www.mydomain.com/sitemap.xml

    The gunzipped version is smaller to download, so you can add that too. But I have never tried GZ version. Why dont you add the sitemap in Google Webmaster Tools and check if it can retrieve and extract the URLs.
    {{ DiscussionBoard.errors[2969291].message }}
  • Profile picture of the author bhanjerikley
    Site link is not in itself a symbol of the REP. Which is only there to help our crawlers find your map. usual rules apply EPR, then you can block robots from indexing your site.
    {{ DiscussionBoard.errors[2981354].message }}

Trending Topics