12 {{ upvoteCount | shortNum }}
12 {{ upvoteCount | shortNum }}

Robots.txt file help !

by rahulkum
10 replies
Hello,
I created the robots.txt file like this(actually it was automatically created with the help of seo by yoast plugin) -
User-agent: *
Disallow: /wp-admin/

But when I checked my robots.txt file by writing in the browser(www.mysite.com/robots.txt),it looked like this -
User-agent: * Disallow: /wp-admin/

It was written in just one single line.I think it might create some problem while crawling and indexing.How could I fix it?
Please Help.
#file #robotstxt
  • Profile picture of the author nmwf
    That robots.txt file is a plain text file, so you can overwrite it with a correctly formatted version via FTP. Here's a guide if you need it. How to edit robots.txt in WordPress
    Signature
    Write comprehensible articles on *any* topic in seconds with First Draft...
    First Draft's: Download | Add-Ons | Templates | Purchase | Support | Affiliates
    {{ DiscussionBoard.errors[10204761].message }}
  • Profile picture of the author totalprocessing.net
    just change it with ftp access, though it probably doesn't matter because the engine will read left to right like we do and will figure it out.
    Signature
    Premium WooCommerce & WordPress Plugins $10/ea. or FREE With Membership. PluginForage.com
    {{ DiscussionBoard.errors[10204832].message }}
    • Profile picture of the author rahulkum
      I did it with the help of ftp client also.When i open the robots.txt file,it's nicely arranged line by line-
      User-agent: *

      Disallow: /cgi-bin/

      Disallow: /wp-admin/

      Disallow: /discount/

      Disallow: /comments/feed/

      Disallow: /trackback/

      Disallow: /index.php

      Disallow: /xmlrpc.php

      Disallow: /wp-content/plugins/



      User-agent: NinjaBot

      Allow: /



      User-agent: Mediapartners-Google*

      Allow: /



      User-agent: Googlebot-Image

      Allow: /wp-content/uploads/

      but when i write it in the url to verify i get something like this-

      User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /discount/ Disallow: /comments/feed/ Disallow: /trackback/ Disallow: /index.php Disallow: /xmlrpc.php Disallow: /wp-content/plugins/ User-agent: NinjaBot Allow: / User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/

      How should I fix it?
      {{ DiscussionBoard.errors[10204887].message }}
      • Profile picture of the author nmwf
        Originally Posted by rahulkum View Post

        but when i write it in the url to verify i get something like this-

        User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /discount/ Disallow: /comments/feed/ Disallow: /trackback/ Disallow: /index.php Disallow: /xmlrpc.php Disallow: /wp-content/plugins/ User-agent: NinjaBot Allow: / User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/
        If you try to access any one of those urls with Google fetch (https://support.google.com/webmaster.../6066468?hl=en), what happens?

        For instance, if you try to load http://www.yourwebsite.com/discount/, what do you see? If you're denied access, I wouldn't worry about it, as the txt file is working like it's supposed to. If you see the page's content, then you'll probably need to contact your host provider for help.
        Signature
        Write comprehensible articles on *any* topic in seconds with First Draft...
        First Draft's: Download | Add-Ons | Templates | Purchase | Support | Affiliates
        {{ DiscussionBoard.errors[10204899].message }}
        • Profile picture of the author rahulkum
          So when i go to this - mysite.com/discount, i land on the homepage of the site.
          {{ DiscussionBoard.errors[10204925].message }}
          • Profile picture of the author totalprocessing.net
            Originally Posted by rahulkum View Post

            So when i go to this - mysite.com/discount, i land on the homepage of the site.
            Great. You are good to go then.
            Signature
            Premium WooCommerce & WordPress Plugins $10/ea. or FREE With Membership. PluginForage.com
            {{ DiscussionBoard.errors[10215369].message }}
      • Profile picture of the author rwil02
        Originally Posted by rahulkum View Post

        ...
        but when i write it in the url to verify i get something like this-

        User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /discount/ Disallow: /comments/feed/ Disallow: /trackback/ Disallow: /index.php Disallow: /xmlrpc.php Disallow: /wp-content/plugins/ User-agent: NinjaBot Allow: / User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/

        How should I fix it?
        That's almost certainly just a rendering issue in the browser.
        HTML treats multiple whitespace (tabs, spaces, carriage returns and new lines) as a single character.

        So if your web server is returning a type of text/html instead of text/plain, or you have a "bad" browser, it will render it out as HTML layout, and collapse all your whitespace for you.

        Right clicking the page and choosing "View Source" should give you a better view of the actual response from the server.
        Signature

        Roger Willcocks
        L-Space Design
        Please vote to help me win a 3kW solar array

        {{ DiscussionBoard.errors[10214594].message }}
  • Profile picture of the author totalprocessing.net
    You should be sure to run your site through google mobile test to make sure you haven't blocked anything that google requires to consider your site mobile friendly.
    Signature
    Premium WooCommerce & WordPress Plugins $10/ea. or FREE With Membership. PluginForage.com
    {{ DiscussionBoard.errors[10204912].message }}
    • Profile picture of the author rahulkum
      Originally Posted by totalprocessing.net View Post

      You should be sure to run your site through google mobile test to make sure you haven't blocked anything that google requires to consider your site mobile friendly.
      I got this message after doing the test-
      Awesome! This page is mobile-friendly.
      {{ DiscussionBoard.errors[10204931].message }}
  • Profile picture of the author royaloptimizer
    The robots.txt file defines how a search engine spider like Googlebot should interact with the pages and files of your web site.
    If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. The robots.txt is a very simple text file placed on your web server.
    The robots.txt is a very simple text file that is placed on your root directory. An example would be www.your-domain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index.
    You can ONLY have one robots.txt on your site and ONLY in the root directory.

    robots.txt depends on what you want it to do:
    User-Agent: * Disallow:
    This will allow all bots to all files

    User-Agent: * Disallow: /
    This blocks all bots to all files
    {{ DiscussionBoard.errors[10226044].message }}

Trending Topics