9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

Sitemap contains urls which are blocked by robots.txt

by toffos
8 replies
What should i do im using word press
#blocked #robotstxt #sitemap #urls
  • Profile picture of the author irawr
    Banned
    Originally Posted by toffos View Post

    What should i do im using word press
    Edit robots

    or delete it
    {{ DiscussionBoard.errors[10419679].message }}
    • Profile picture of the author toffos
      how to delete or edit them...if we can edit them how and what need to be edit ?? im a newbie your help appreciated
      {{ DiscussionBoard.errors[10419683].message }}
      • Profile picture of the author irawr
        Banned
        Originally Posted by toffos View Post

        how to delete or edit them...if we can edit them how and what need to be edit ?? im a newbie your help appreciated
        robots.txt .....
        {{ DiscussionBoard.errors[10419694].message }}
  • Profile picture of the author lucknowseocompany
    Goto this plugin WP Robots Txt
    Goto settings/privacy settings
    Then add this to your robot txt file
    User-agent: Googlebot
    Allow: /
    {{ DiscussionBoard.errors[10419881].message }}
  • Profile picture of the author essmeier
    Robots.txt is a text file that sits at the root of your domain. You need to use either a Wordpress plugin or an FTP program to access the file and edit it.

    Charlie
    Signature
    Work Online From Home
    {{ DiscussionBoard.errors[10422102].message }}
  • Profile picture of the author David V
    All the answers are good, but I think backwards for what the question was.

    Whether a blocked url is showing in the sitemap has nothing to do with what's in the robots.txt file.

    You didn't say how you created your sitemap and that makes a difference.

    If your using WordPress and the Yoast SEO plugin, you can STOP those pages/urls from being put into the sitemap in the first place.

    What goes into the sitemap is up to you, and the priority that is set for each listing.

    So how you create the sitemap will depend on how to remove the blocked pages.

    Either way, do not list "those" pages in the sitemap.
    The sitemap will only list what you tell it to list.

    Without more detail I can't tell you the exact steps to correct it, but you'll need to recreate the sitemap with only the urls you want listed.

    Make sense?
    {{ DiscussionBoard.errors[10426122].message }}

Trending Topics