10 {{ upvoteCount | shortNum }}
10 {{ upvoteCount | shortNum }}

what is the difference between robot meta tag and robot.txt?

by matthewalbertson
7 replies
  • SEO
    • |
Hello Guys,

Can you let me know what is the major difference between robot meta tag and robot.txt?
#difference #meta #robot #robottxt #tag
  • Profile picture of the author asiancasanova
    Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, Pandecta Magazine's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider (robot) to ignore it. The Robots META tag does exactly the same thing as the robots.txt file - but it is not as reliable. Not all robots honor the robots meta tag. Use it if your site is in a subdirectory like www.freewebspace.com/users/mycoolhomepage/ and you can't get the server administrator to add (or add changes to) a robots.txt file.
    Signature
    $1k a day?
    Click here
    {{ DiscussionBoard.errors[6624793].message }}
  • Profile picture of the author CyborgX
    Both are same as robots txt is used to stop web crawler not to crawl particular page or content as meta robots work is also same . The main difference is of structure how to use them both. Working of both robots.txt ant meta robots are same.
    Signature
    Best online hotel booking website
    {{ DiscussionBoard.errors[7397753].message }}
  • Profile picture of the author It Is Me
    Originally Posted by matthewalbertson View Post

    Hello Guys,

    Can you let me know what is the major difference between robot meta tag and robot.txt?
    Here you go What's the difference between robots.txt and Meta Tags? | Go Daddy Help | Go Daddy Support
    Signature

    --
    AdRenter.com, AudioVideoGraphics.Com, ChooseHosts.com, CommentSubmitter.com, CommissionGate.com, CustomBacklinks.com, IncredibleLists.com.... Selling On GetTheseDomains.com
    --
    Internet Marketers Need These Products - IMersNeed.com

    {{ DiscussionBoard.errors[7398208].message }}
  • Profile picture of the author mangomedia1
    In meta tags , you have to write particular page follow , no follow tags.but robots used to google bots, write the coding for just crawl my wanted files and folders from server.
    {{ DiscussionBoard.errors[7398933].message }}
  • Profile picture of the author Weblover50
    Both serve the same purpose, and it is up to the site owner to chose whichever is convenient. robot.txt would be easier if you want to block specific directories etc. Meta tag could be better if you have lots of pages to be blocked from different directories / without a pattern. Adding a meta tag in the page itself would be easier in such occassions.
    Signature

    Hosting specials - Hostgator Review and Inmotion Coupon

    {{ DiscussionBoard.errors[7399070].message }}
    • Profile picture of the author Paul240480
      ................ so if I follow this correctly.... I want the bots to crawl all of my pages so a 'simple' robots index,follow metatag is fine?
      Signature

      Paul240480
      http://www.gitessouthbrittany.com

      {{ DiscussionBoard.errors[7403260].message }}
  • Profile picture of the author Elfen Lied
    Nothing will prevent anyone typing in the URI to access the page. If you are concerned about security you need to implement more reliable methods of preventing unauthorised visitors

    Pages blocked by robots.txt may appear in results as PIPs (Partially Indexed Pages), which shows a URL only if links are found to the page.
    With a "noindex" directive in the robots meta this should not occur, as bots can read the page content and the directive for not showing the page in the index.
    {{ DiscussionBoard.errors[7628387].message }}

Trending Topics