Need Help about robots.txt file inclusion

by benthomas642 4 replies
Hi to All, I m just created robots.txt file for my website
And I updated at root file directory on the web server.
I used following codes in the robots.txt file which are:
User-agent: *

So I added these above codes and updated on the web server. But on Google Search Console, it has been showing me the old codes not the new codes above I updated. Can anyone tell me why search console doesn't showing me the updated code that I used. Or Google search console takes some times to updated the data. Is the above codes I used were wrong?

Tell me why or suggest me what should I have to do.
Thanks a lot.
#search engine optimization #file #inclusion #robotstxt
Avatar of Unregistered
  • Profile picture of the author TriState Technology
    {{ DiscussionBoard.errors[11074715].message }}
    • Profile picture of the author benthomas642
      Thanks I already used the same. And it's working.
      {{ DiscussionBoard.errors[11075849].message }}
  • Profile picture of the author GoClickOn
    The simplest robots.txt file uses two key words, User-agent and Disallow. User-agents are search engine robots, most user-agents are listed in the Web Robots Database. Disallow is a command for the user-agent that tells it not to access a particular URL. On the other hand, to give Google access to a particular URL that is a child directory in a disallowed parent directory, then you can use a third key word, Allow.
    {{ DiscussionBoard.errors[11074733].message }}
  • Profile picture of the author paulgl
    How can google crawl your emails? Just asking....

    99% of the time, the best robots.txt is blank.


    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[11074806].message }}
    • Profile picture of the author benthomas642
      I don't thinks so about how can we use just a blank. If we use blank file or we don't it means that search engine is crawling your entire website links including those link that shouldn't be accessible from the search engine.
      When you type on Google. Then you can see that all links would be crawled including those that is not necessary needed to be crawled on Search engine.
      {{ DiscussionBoard.errors[11075847].message }}
Avatar of Unregistered

Trending Topics

  • 11 {{ upvoteCount | shortNum }}

    How Do Highly Successful People Stay Positive?

    Hi Gang, I just stumbled upon this great article and wanted to share! 6 Ways Highly Successful People Stay Positive So the question begs, how do YOU stay positive? What ... [read more]

  • 16 {{ upvoteCount | shortNum }}

    Free software that can record more that 10 minutes ?

    spartan14 in Internet Marketing

    Hy all i am more a youtube guy and i plan to make a course for my subscribers with some video series and i need to record videos longer that ... [read more]

  • 218 {{ upvoteCount | shortNum }}

    What is the main reason for the 95% failure rate in internet marketing?

    Here is my take on the matter: Look, a lot of people get into the internet marketing business only because of its huge money-making potential without having a love for ... [read more]

  • 8 {{ upvoteCount | shortNum }}

    troubleshooting sales

    sparky63 in Internet Marketing

    hi peeps i have just created my first website about premium plr ebook course where you can rebrand and sell i have had quite abit of traffic over the ... [read more]

  • 13 {{ upvoteCount | shortNum }}

    How to make money online as an 18 year-old?

    I have a real job already, but i also want to have a steady passive income from the internet...any suggestions how ?