by rihsam
16 replies
  • SEO
  • |
I am having a serious problem with my blog's mobile version. When i take Google's mobile friendly test it says "The requested URL redirected to How Cancer Kills which is disallowed by robots.txt." Please help me to fix this issue.

Thank you

Regards
Rihsam
#error #seo
  • Profile picture of the author SEO-Dave
    Originally Posted by rihsam View Post

    I am having a serious problem with my blog's mobile version. When i take Google's mobile friendly test it says "The requested URL redirected to How Cancer Kills which is disallowed by robots.txt." Please help me to fix this issue.

    Thank you

    Regards
    Rihsam
    Edit your robots.txt file:

    http://www.howcancerkills.com/robots.txt

    Code:
    User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Disallow: /?m=0 Disallow: /?m=1 Allow: / Sitemap: http://howcancerkills.com/atom.xml?redirect=false&start-index=1&max-results=500
    That's a mess and isn't doing what you think, for example you are disallowing your categories (they are under the URLs /search/) which means Google doesn't index them which means they aren't passing SEO link benefit through the site!!!

    I recommend deleting the content of the entire file, save it and upload so you have a blank robots.txt file.

    Your robots.txt file is a good example how to damage your own SERPs!

    If there's sections of the site you want to disallow what are they and why?

    David
    {{ DiscussionBoard.errors[9704238].message }}
  • Profile picture of the author SEO-Dave
    What's really funny (from an SEO perspective) is the site has the useful categories blocked and the almost useless dated archives indexable!

    Where ever you got your SEO info from, stay away from that source.

    David
    {{ DiscussionBoard.errors[9704247].message }}
  • Profile picture of the author Granulr
    If i were you, i would delete your robots file and start again.

    Are you using any sort of CMS like Wordpress?
    {{ DiscussionBoard.errors[9705478].message }}
  • Profile picture of the author SEO-Dave
    Why are you still disallowing /search?

    Those are your categories, you are shooting your site in the foot SEO wise.

    Your current robots.txt file is SEO damaging, there is nothing useful in it, delete it.

    Are you using the blogger custom robots.txt options https://support.google.com/blogger/answer/2472627?hl=en Can you remove /search? Can you use an empty robots.txt file?

    David
    {{ DiscussionBoard.errors[9714137].message }}
    • Profile picture of the author rihsam
      Dear Dave

      According to the guideline given by the various blogs, SEO experts and blogger experts, Disallow: /search means instructing bots not to search certain pages in the blog like labels, archives, tags. etc.
      Example: How Cancer Kills: Breast cancer

      I may be wrong according to your view, Please refer these articles and instruct me for further changes. I appreciate your help.
      Add Custom Robots.txt file in Blogger - Blogger Tips and Tricks
      Add Custom Robots.txt File in Blogger | Helplogger

      thank you
      {{ DiscussionBoard.errors[9716302].message }}
      • Profile picture of the author SEO-Dave
        Originally Posted by rihsam View Post

        Dear Dave

        According to the guideline given by the various blogs, SEO experts and blogger experts, Disallow: /search means instructing bots not to search certain pages in the blog like labels, archives, tags. etc.
        Example: How Cancer Kills: Breast cancer

        I may be wrong according to your view, Please refer these articles and instruct me for further changes. I appreciate your help.
        Add Custom Robots.txt file in Blogger - Blogger Tips and Tricks
        Add Custom Robots.txt File in Blogger | Helplogger

        thank you
        First your description of what the disallow does is wrong, it stops search engines completely indexing any URL /search/... Disallowing a webpage (means the same as noindex,nofollow) takes it out of the PR flow cycle completely, which means any links you send to a webpage that is disallow will be wasted, the PR will be completely lost.

        Look at your home page and:
        Count how many links you have in total (rough estimate will be fine)
        Count how many links go to webpage starting /search/..

        What's the rough percentage of /search/.... because that's the percentage of PR/link benefit you are loosing with your current robots.txt file.!

        I didn't count them, but looks like you could have more /search/... links than none /search/... that's a lot of lost link benefit!!!

        If 50% of your links go to /search/... you are wasting 50% of your internal PR, All that hard earned PR/link benefit from building backlinks wasted because of one bad robots/txt rule!

        As a Blogger user the /search/ pages are your category archives, not actual search results: you are manually setting with posts are found in one of the searches (it's a category archive).

        As long as your category archives use an excerpt of the posts (which it does on your site) there are no duplicate content issues to be concerned about.

        You are seriously damaging your sites rankings with that robots.txt file.

        If you wanted the /search/... pages not indexed, but not take them completely out of the PR flow you would add a noindex,follow robots meta tag only to the webpages with URLs starting /search/... this would stop search engines indexing the categories (if you think it's duplicates content, it's not) and still allow PR to flow through the /search/.. pages to your posts. The noindex still wastes some PR, overall ~15% of the PR you are currently wasting would still be wasted, so this would recover ~85% of what you are loosing right now and have the same impact on not indexing /search/...

        Take my advice as someone who knows more about SEO than any of the sites you've taken advice on this, delete your robots.txt file completely (or make it blank) and don't add the noindex,follow robots meta tag. Allow search engines to index it all.

        Want to improve your site, reduce the number of /search/... (categories) webpages: if you can't add half a dozen posts to a category, don't create it, delete all single keyword categories, aim for a minimum 2+ keyword categories like "Breast Cancer Research". Remove the monthly archive widget from your site completely, they serve no purpose.

        David
        {{ DiscussionBoard.errors[9716573].message }}
  • Profile picture of the author webcomincorp
    go into browser type your url /robots.txt . find the disallow text there. and change it to allow
    {{ DiscussionBoard.errors[9714140].message }}
  • Profile picture of the author SEO-Dave
    Looked at one of the articles you linked to which says this:

    • User-agent: *
    This is for all robots marked with asterisk (*).
    That tells use the person who wrote the article doesn't know shit about SEO.

    The * is a wild card, no search engine spider is marked with a *. That makes no sense.

    What User-agent: * really means is ALL search engine spiders follow the next set of rules.

    This is what a robots.txt file should look like:

    http://stallion-theme.co.uk/robots.txt

    Code:
    User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /go/
    I want all search engine spiders (*) to not includes anything under these three directories:
    wp-admin and wp-includes are WordPress admin directories, there's nothing under those folders I want indexing.

    The go folder holds my affiliate links, I use a PHP script to link to affiliate sites using a 302 redirected webpage and I don't want those webpages indexed by Google etc...

    All my categories are indexed by all search engines.

    David
    {{ DiscussionBoard.errors[9716599].message }}
    • Profile picture of the author BrownBeard
      Originally Posted by SEO-Dave View Post

      That tells use the person who wrote the article doesn't know shit about SEO.
      Man I've never heard o' somebody havin' so many problems with robots.txt
      Signature

      I be a t' pirate captain Brown Beard

      {{ DiscussionBoard.errors[10636595].message }}
  • Profile picture of the author xnaveed93gb

    This is our latest design pakistan dress design site India Pakistani,mobile phone health beauty cosmetics fashion designs and much more is available.
    {{ DiscussionBoard.errors[10549901].message }}
  • Profile picture of the author xnaveed93gb
    Free India Actress Wallpapers Latest Priyanka Chopra Pictures very cute and beautyfull wallpapers.Free India Actress Wallpapers Priyanka Chopra Wallpaper very cute and beautyfull wallpapers.
    {{ DiscussionBoard.errors[10636561].message }}

Trending Topics