Need tool which does this

by 3 replies
4
Crawl a site, obeying NoIndex/NoFollow and show me exactly like Google would index the page, eg. pages which are NoIndex/NF in different colors.

G.
#search engine optimization #tool
  • I dont know about "exactly like googlebot" but Xenu Link Sleuth is good for crawling.
  • yes i have xenu, but not sure whether it does that.
    • [1] reply
    • Not sure what you mean by obeying nofollow. That has nothing
      to do with crawling or indexing. Your pages would not be nofollow.
      The links to the pages could be nofollow, but that does not keep
      them from being indexed. Only noindex will. If a page is on the web,
      and does not have the noindex, it could be indexed.

      Paul
Join the discussion

Next Topics on Trending Feed