8 {{ upvoteCount | shortNum }}
8 {{ upvoteCount | shortNum }}

Robots.txt file

by rahulkum
7 replies
  • SEO
    • |
Hello,
I created the robots.txt file like this(actually it was automatically created with the help of seo by yoast plugin) -
User-agent: *
Disallow: /wp-admin/

But when I checked my robots.txt file by writing in the browser(www.mysite.com/robots.txt),it looked like this -
User-agent: * Disallow: /wp-admin/

It was written in just one single line.I think it might create some problem while crawling and indexing.How could I fix it?
Please Help.
#file #robotstxt
  • Profile picture of the author SEOwarez
    its probably just the newline formating. when you create a textfile and you hit the "enter" key it will be stored as "\n" and you wont see that when you open it in the browser or a text editor, but its there.

    now operating systems treat newlines differently. for example on windows you have "\r\n" and on linux its just "\n" that means a file created on linux (like on your server probably) might show up on windows as if it would be all in 1 line. however thats not the case, when a program reads the file it will still notice the "\n" character as newline. so googlebot should properly see the robots.txt, but to be 100% sure, just login to your server through SSH and open the robots.txt in a texteditor such as "vim" then you will know for sure if it has proper formating
    Signature
    {{ DiscussionBoard.errors[10238539].message }}
  • Profile picture of the author nettiapina
    Robots.txt should have line changes as far as I know. Are you sure it's displaying correctly for you? The link in your post, and the robots.txt of the site in your profile are both ok.
    Signature
    Links in signature will not help your SEO. Not on this site, and not on any other forum.
    Who told me this? An ex Google web spam engineer.

    What's your excuse?
    {{ DiscussionBoard.errors[10238633].message }}
  • Profile picture of the author disseomelbourne
    just hit Enter. put it in new line.
    otherwise its alright.
    {{ DiscussionBoard.errors[10240408].message }}
  • Profile picture of the author Hemanth Malli
    Just you place that code in a new line, then the problem was solved.
    Signature
    Inverselogix
    CEO
    {{ DiscussionBoard.errors[10240459].message }}
  • Profile picture of the author kemilus
    The robots.txt file gives guidelines to robotized web crawlers going by your website, including search crawlers. if you set the file as disallow, google will not scrape your site, if allow, google will find it, so if there is something that you do not want google to get it, set it like this. besides, you can use some tools to help you, go whitehatbox
    {{ DiscussionBoard.errors[10254968].message }}
  • Profile picture of the author seothink
    It's using for secure with allow to google crawler to cache and index into their search page .
    {{ DiscussionBoard.errors[10257062].message }}
  • Profile picture of the author jasminemartin
    Yup robots.txt file not useful if codes are on one line correct this.
    {{ DiscussionBoard.errors[10268912].message }}

Trending Topics