8 {{ upvoteCount | shortNum }}
8 {{ upvoteCount | shortNum }}

Question on Robots,txt file

by vikpathania
7 replies
  • SEO
    • |
Here is a question for you guys:

If I add following syntax:

User-agent: *
Disallow: /

This means it will disallow all the bots to crawl any of my page. Right?

But, would it also stop crawler from crawling my root page? Or crawler can still crawl my root page?


Looking forward for great views.
#file #online marketing #question #robots #robots.txt #seo #txt
  • Profile picture of the author imsirigiri
    Search engine crawlers first check the robots.txt and .htaccess files. So, when you specify the above code and request the crawlers to not index your site, reputed search engine crawlers obey that and do not index even your root folder files.

    Hope this clarifies.
    Signature
    Need a Technical Support VA on an Hourly Basis? || Need AdSense Microniche Sites Research and Development? PM me.
    {{ DiscussionBoard.errors[11001069].message }}
    • Profile picture of the author vikpathania
      Originally Posted by imsirigiri View Post

      Search engine crawlers first check the robots.txt and .htaccess files. So, when you specify the above code and request the crawlers to not index your site, reputed search engine crawlers obey that and do not index even your root folder files.

      Hope this clarifies.
      Are you sure that it will not crawl the root? I am still bit confused on this. if we type /abc that means crawler will not crawl abc page.. But, it will crawl the root page and all other pages. So, if we use /, then it will not crawl the page address after / means the inner pages. How it disable crawler from not crawling the root.


      Might be confusing, sorry for bad English....
      {{ DiscussionBoard.errors[11001124].message }}
      • Profile picture of the author imsirigiri
        It doesn't.

        Originally Posted by vikpathania View Post

        Are you sure that it will not crawl the root? I am still bit confused on this. if we type /abc that means crawler will not crawl abc page.. But, it will crawl the root page and all other pages. So, if we use /, then it will not crawl the page address after / means the inner pages. How it disable crawler from not crawling the root.


        Might be confusing, sorry for bad English....
        Signature
        Need a Technical Support VA on an Hourly Basis? || Need AdSense Microniche Sites Research and Development? PM me.
        {{ DiscussionBoard.errors[11001131].message }}
        • Profile picture of the author vikpathania
          Originally Posted by imsirigiri View Post

          It doesn't.
          Ok. I'll check
          {{ DiscussionBoard.errors[11001196].message }}
  • Profile picture of the author avemfly619619
    can anyone explain about Sitemap.xml and sitemap.html?
    it is necessary upload .html page in website.
    Signature
    digital marketing company in Delhi | seo company in Delhi | SMO Company in Delhi
    {{ DiscussionBoard.errors[11001102].message }}
  • Profile picture of the author imsirigiri
    Both are not mandatory.

    Sitemap.XML helps the crawlers to understand how a website is structured and enables them to find out deep linked pages and resources.

    Sitemap.HTML is for website visitors who want to understand where they need to go in order to read or understand more.

    It is recommended to use sitemap.xml but sitemap.html is beneficial if there are multiple services or categories which may require a little headsup for the visitors.
    Signature
    Need a Technical Support VA on an Hourly Basis? || Need AdSense Microniche Sites Research and Development? PM me.
    {{ DiscussionBoard.errors[11001107].message }}
  • Profile picture of the author Fahad Nazir
    This command instructs the robot not to visit any of the page on the site.. because it applies to all the pages.
    {{ DiscussionBoard.errors[11001261].message }}
  • Profile picture of the author kodeforest
    Banned
    [DELETED]
    {{ DiscussionBoard.errors[11001276].message }}

Trending Topics