15 {{ upvoteCount | shortNum }}
15 {{ upvoteCount | shortNum }}

Robots and Meta Robots

by james7
14 replies
  • SEO
    • |
What is the difference between robots.txt and meta robots? Which one is more beneficial.
#meta #robots
  • Profile picture of the author majorcheaphazard
    robots.txt instruct robots, specially search engine robots on how to crawl & index pages on a website, however, malicious crawlers are most likely ignoring it.

    meta robots on the other hand work in a different way, for example, a meta noindex tells engines they can visit but they are not allowed to display the URL in results.
    {{ DiscussionBoard.errors[6499088].message }}
  • Profile picture of the author laqzya2012
    robots.txt -The robots.txt file is a simple text file that resides within the root directory of a website. The purpose of this file is to help search engines understand which content you would like them to illustrate to users within the search engine result pages and which pages they should ignore.

    Meta robots - The meta robots field within the header of your source code serves a similar purpose to that of the robots.txt file
    {{ DiscussionBoard.errors[6518714].message }}
    • Profile picture of the author sadeeshbala
      As far as my concern - robots.txt is beneficial
      {{ DiscussionBoard.errors[6518791].message }}
  • Profile picture of the author webdevpro
    Robots.txt can stop search engines to fetch the restricted contents but on the other side and to obey the meta robot tag first they need to fetch the contents at least and this will be counted towards bandwidth usage.
    Signature
    {{ DiscussionBoard.errors[6520779].message }}
  • Profile picture of the author d10ko
    Hello, i had market not allow google bot to index... on wordpress, then i add the site in webmaster tool and when submitted a sitemap it couldn't access because of robots.txt. then i changed the option to allow googlebot to index... added robots.txt and when fetched as googlebot there was restriction again. please tell me what to do to make it well-done. here is the screenshots of filezila (where i have uploaded a robots.txt file) and webmaster tools where the field of allow/disallow is empty.
    {{ DiscussionBoard.errors[6545628].message }}
    • Profile picture of the author vgvetter
      Originally Posted by d10ko View Post

      Hello, i had market not allow google bot to index... on wordpress, then i add the site in webmaster tool and when submitted a sitemap it couldn't access because of robots.txt. then i changed the option to allow googlebot to index... added robots.txt and when fetched as googlebot there was restriction again. please tell me what to do to make it well-done. here is the screenshots of filezila (where i have uploaded a robots.txt file) and webmaster tools where the field of allow/disallow is empty.

      Here's what worked for us:

      Site Robots.txt File
      Generate a site robots.txt file as shown in bold below with notepad or the Google robots.txt tool and upload via FTP into the site root directory.

       User-agent: *
      Allow: /       User-agent: Googlebot Allow: /

      sitemap: http://www.mysite.com/sitemap.xml

      In the event the site uses Google AdSense, add the following:

      User-agent: Mediapartners-Google
      Disallow:

      Upload the above robots.txt file via FTP into the site root directory. From the browser if you now       type in www.mysite.com/robots.txt, the file shown in bold above should display.

      We excluded the last two lines User agent Mediapartners thru Disallow, and are using AdSense without problem,
      Signature
      Websites with WordPress
      Free eBook: Download; Backup; Navigation; Forms; Graphics; RSS; Traffic; SEO; Customization; News Letters; eCommerce
      {{ DiscussionBoard.errors[6548883].message }}
  • Profile picture of the author gPlayer
    What is the difference between robots.txt and meta robots?
    Robot.txt is a text file and meta robot is HTML file. Robot.txt stored in main directory and meta robot tag create page level instruction. Robot.txt file is used for which bot are not allowed for which pages. You can use any of them. Robot.txt is much better.
    Signature
    {{ DiscussionBoard.errors[6546758].message }}
  • Profile picture of the author stopper
    Meta robots and robots seem to play similar roles with different instructions for each bot.
    Signature
    http://www.charlesmomo.com Are you interested in 200 to 400 visitors a day to your site
    {{ DiscussionBoard.errors[6549243].message }}
  • Profile picture of the author ankushkohli
    Robots.txt is a text file which search engine spiders read in-order to interact with indexing your content. You can allow or disallow files/folders indexing through this.


    The Meta robots tag creates page level instructions for search engine bots.
    Signature
    Orange County Local SEO Company | Emarketing Guide
    {{ DiscussionBoard.errors[10609121].message }}
  • Profile picture of the author wdcbangalore
    robots.txt is used to ignore the particular content while google spider crewel the web pages and Meta robot used to allow the google spider to crawl but in not display the url.
    Signature
    Best Website Design Company Bangalore | Top website design company in bangalore | Bangalore Web Design Company
    {{ DiscussionBoard.errors[10609160].message }}
  • Profile picture of the author vedasri
    There are some important differences between robots.txt and meta 'noindex' from an indexing perspective.

    1. Robots.txt won't stop a URL being displayed in Google serps.

    2. meta 'noindex' will stop a URL being displayed in Google serps.
    {{ DiscussionBoard.errors[10620732].message }}
  • Profile picture of the author yukon
    Banned


    Your next question should be: How many years has nofollow forum sig. links been useless for SEO?
    {{ DiscussionBoard.errors[10620740].message }}
  • Profile picture of the author Serem3750
    [DELETED]
    {{ DiscussionBoard.errors[10620754].message }}
  • Profile picture of the author ziyapathan
    The Robots Exclusion Protocol (REP) is a group of web standards that regulate web robot behavior and search engine indexing,The Meta tag option tells bots not to index individual files, whereas Robots.txt can be used to restrict access to entire directories.
    There are two important considerations when using the robots <META> tag:

    robots can ignore your <META> tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
    the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (perhaps on some other site), and so still arrives at your undesired page.
    Signature
    Hire Mobile Developer||Apache Cordova Developers|Xamarin developer india
    {{ DiscussionBoard.errors[11012433].message }}
  • Profile picture of the author tyagi
    The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

    Thank you!

    Happy to Help you.
    {{ DiscussionBoard.errors[11012599].message }}

Trending Topics