3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

robots.txt issue when submitting my website's sitemap

by Vikram Anand
3 replies
  • SEO
    • |
I had submitted the sitemap for my website on Google Search Console. However I am getting the following error message:
"Sitemap contains urls which are blocked by robots.txt"
When I check site:mydomain, I find that Google has indexed my site pages. I had enabled robots noindex, nofollow for some pages, categories and tags.
Those are not shown when I run this command site:mydomain in google search.
However in the Search console in the warnings for sitemaps, I see the value as mydomain itself.
As I am not really an expert in handling robots.txt issues, can anyone help me to understand, what is happening? Is my site getting crawled by Google (except those pages which I don't want). If there is any problem, can anyone help me to resolve it please...
#issue #robotstxt #sitemap #submitting #website
  • Profile picture of the author Peter Stavrou
    Are you using SEO By Yoast to generate that sitemap by any chance?

    Also what URL's are being blocked exactly?
    Are they the same pages that you enabled robots noindex, nofollow on?
    Signature

    Online Business Made Simple!
    http://www.PeterStavrou.com

    {{ DiscussionBoard.errors[10599844].message }}
    • Profile picture of the author Vikram Anand
      Originally Posted by Peter Stavrou View Post

      Are you using SEO By Yoast to generate that sitemap by any chance?

      Also what URL's are being blocked exactly?
      Are they the same pages that you enabled robots noindex, nofollow on?
      Yes. Peter..I am using SEO by Yoast for my wordpress site.
      In Search Console, 3 urls are shown as being blocked viz. my domain and 2 other pages.

      These are not the pages for which I have enabled robots noindex, nofollow.
      I have enabled robots for privacy policy page and categories, tags (taxonomies in SEO by Yoast)
      However when I check site:mydomain on google, I see all my pages have been indexed (except the ones for which I have enabled Robots)

      Can you guide me on this? If there is any problem how to resolve it?
      Signature
      Is there any area of your life where your need help??

      Click Here to access tons of resources on Self Help
      {{ DiscussionBoard.errors[10600054].message }}
  • Profile picture of the author Vikram Anand
    The issue has been resolved. When I was creating the sitemap, that url for which I had enabled robots no index was also getting added.
    When I removed that url from sitemap, the error was resolved.
    Signature
    Is there any area of your life where your need help??

    Click Here to access tons of resources on Self Help
    {{ DiscussionBoard.errors[10601453].message }}

Trending Topics