Robots.txt file help !

by 10 replies
12
Hello,
I created the robots.txt file like this(actually it was automatically created with the help of seo by yoast plugin) -
User-agent: *
Disallow: /wp-admin/

But when I checked my robots.txt file by writing in the browser(www.mysite.com/robots.txt),it looked like this -
User-agent: * Disallow: /wp-admin/

It was written in just one single line.I think it might create some problem while crawling and indexing.How could I fix it?
Please Help.
#programming #file #robotstxt
  • That robots.txt file is a plain text file, so you can overwrite it with a correctly formatted version via FTP. Here's a guide if you need it. How to edit robots.txt in WordPress
  • just change it with ftp access, though it probably doesn't matter because the engine will read left to right like we do and will figure it out.
    • [1] reply
    • I did it with the help of ftp client also.When i open the robots.txt file,it's nicely arranged line by line-
      User-agent: *

      Disallow: /cgi-bin/

      Disallow: /wp-admin/

      Disallow: /discount/

      Disallow: /comments/feed/

      Disallow: /trackback/

      Disallow: /index.php

      Disallow: /xmlrpc.php

      Disallow: /wp-content/plugins/



      User-agent: NinjaBot

      Allow: /



      User-agent: Mediapartners-Google*

      Allow: /



      User-agent: Googlebot-Image

      Allow: /wp-content/uploads/

      but when i write it in the url to verify i get something like this-

      User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /discount/ Disallow: /comments/feed/ Disallow: /trackback/ Disallow: /index.php Disallow: /xmlrpc.php Disallow: /wp-content/plugins/ User-agent: NinjaBot Allow: / User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/

      How should I fix it?
      • [2] replies
  • You should be sure to run your site through google mobile test to make sure you haven't blocked anything that google requires to consider your site mobile friendly.
    • [1] reply
    • I got this message after doing the test-
      Awesome! This page is mobile-friendly.
  • The robots.txt file defines how a search engine spider like Googlebot should interact with the pages and files of your web site.
    If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. The robots.txt is a very simple text file placed on your web server.
    The robots.txt is a very simple text file that is placed on your root directory. An example would be This file tells search engine and other robots which areas of your site they are allowed to visit and index.
    You can ONLY have one robots.txt on your site and ONLY in the root directory.

    robots.txt depends on what you want it to do:
    User-Agent: * Disallow:
    This will allow all bots to all files

    User-Agent: * Disallow: /
    This blocks all bots to all files
Join the discussion

Next Topics on Trending Feed