Can any one check my robots.txt file ?

4 replies
  • SEO
  • |
hi
i have added code in my website and want to ensure, everything alright

i am using magneto cms

https://www.eshtari.me/robots.txt

# # Robots.txt for Eshtari.me # # GENERAL SETTINGS # Enables robots.txt rules for all crawlers User-agent: * # # Eshtari sitemap: URL to your sitemap file in Magento directory Sitemap: https://www.eshtari.me/sitemap.xml # # Do not allow indexing files and folders that are required during development: CVS, SVN directory and dump files Disallow: / CVS Disallow: / *. Svn $ Disallow: / *. Idea $ Disallow: / *. Sql $ Disallow: / *. Tgz $ # # GENERAL SETTINGS For MAGENTO # # Do not index the page Magento admin Disallow: / admin / # # Do not index the general technical Magento directory Disallow: / app / Disallow: / downloader / Disallow: / errors / Disallow: / includes / Disallow: / lib / Disallow: / pkginfo / Disallow: / shell / Disallow: / var / # # Do not index the shared files Magento Disallow: / api.php Disallow: / cron.php Disallow: / cron.sh Disallow: / error_log Disallow: / get.php Disallow: / install.php Disallow: / LICENSE.html Disallow: / LICENSE.txt Disallow: / LICENSE_AFL.txt Disallow: / README.txt Disallow: / RELEASE_NOTES.txt # # MAGENTO SEA IMPROVEMENT # # Do not index the page subcategories that are sorted or filtered. Disallow: / *? Dir * Disallow: / *? Dir = desc Disallow: / *? Dir = asc Disallow: / *? Limit = all Disallow: / *? Mode * # # Do not index the second copy of the home page (example.com / index.php /). Un-comment only if you have activated Magento SEO URLs. # # Disallow: / index.php / # # Do not index the link from the session ID Disallow: / *? SID = # # Do not index the page checkout and user account Disallow: / checkout / Disallow: / onestepcheckout / Disallow: / customer / Disallow: / customer / account / Disallow: / customer / account / login / # # Do not index the search page and CEO, non-optimized link categories Disallow: / catalogsearch / Disallow: / catalog / product_compare / Disallow: / catalog / category / view / Disallow: / catalog / product / view / # # Server Settings # # Do not index the general technical directories and files on a server Disallow: / cgi-bin / Disallow: / cleanup.php Disallow: / apc.php Disallow: / memcache.php Disallow: / phpinfo.php # # SETTINGS Image indexing # # Optional: If you do not want to Google and Bing to index your images # User-agent: Googlebot-Image # Disallow: / # User-agent: msnbot-media # Disallow: /
#check #file #robotstxt
  • Profile picture of the author imsirigiri
    Looks good. Do you have any particular question in mind why you need a second opinion on robots.txt file?
    Signature
    Need a Technical Support VA on an Hourly Basis? || Need AdSense Microniche Sites Research and Development? PM me.
    {{ DiscussionBoard.errors[11002107].message }}
  • Profile picture of the author techugo
    It's Look Good. For more information go through the link - https://support.google.com/webmaster.../6062596?hl=en
    {{ DiscussionBoard.errors[11002121].message }}
  • Profile picture of the author vikpathania
    It is all ok.. Anything else do you want to know?
    {{ DiscussionBoard.errors[11002139].message }}
  • Profile picture of the author hariandro
    Yes google will check your robot.txt file..Actually that files consist of which are the data that should not be crawl by google..By accessing that only google got to know which should not be crawl..
    {{ DiscussionBoard.errors[11002427].message }}

Trending Topics