Which is the best option Meta Robot tags or robots.txt?

5 replies
  • SEO
  • |
Hi friends help me to know Which is the best option Meta Robot tags or robots.txt?
#meta #option #robot #robotstxt #tags
Avatar of Unregistered
  • Profile picture of the author fastestseo
    Can you please explain where do you want to use this because robots.txt allow and disallow engines from crawling certain sections of the site, and if want to block a specific page on your website from crawling and indexing it in its search results pages then it is best practice to use a Meta robots tag.
    eFeeders Tech - Providing Advisory in Following:
    Local SEO | Digital Marketing | Website Development
    {{ DiscussionBoard.errors[11465079].message }}
  • Profile picture of the author PIS SLGX
    if you want to deindex a page or directory from Google's Search Results then we suggest that you use a "Noindex" meta tag rather than a robots.txt directive as by using this method the next time your site is crawled your page will be deindexed, meaning that you won't have to send a URL removal request. However, you can still use a robots.txt directive coupled with a Webmaster Tools page removal to accomplish this.

    Using a meta robots tag also ensures that your link equity is not being lost, with the use of the 'follow' command.

    Robots.txt files are best for disallowing a whole section of a site, such as a category whereas a meta tag is more efficient at disallowing single files and pages. You could choose to use both a meta robots tag and a robots.txt file as neither has authority over the other, but "noindex" always has authority over "index" requests.
    {{ DiscussionBoard.errors[11468244].message }}
    • Profile picture of the author smkghosh
      Certainly yes.Noindex has the greatest priority.

      SEO & Blogger at Tekkibytes

      {{ DiscussionBoard.errors[11468647].message }}
  • Profile picture of the author paulgl
    Come on people. It's about crawling vs. indexing.

    If you don't know that, well know you do.

    noindex means don't index.

    robots.tx instructs crawling.

    There is a difference between crawling and indexing.

    All search engines will want to crawl and index by default.

    If that's for you, leave the robots.txt empty and use no metas.

    noindex in a meta can be futile. Google may crawl it from some other source.

    So you instruct not to crawl certain pages.

    The only way to not have a page show up, is to take it offline or behind a password.

    Again, to end the lesson..... robots.txt directs crawling. Noindex tells to not index the page.

    If you have a page that you don't want indexed, take it offline. Otherwise, leave everything alone.


    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[11468489].message }}
  • Profile picture of the author jeevanfp
    Meta Robot tag is better as it helps in forcing the search engine crawlers not to index and display the hidden pages in your server
    {{ DiscussionBoard.errors[11469394].message }}
Avatar of Unregistered

Trending Topics