I checked some of my competitors robots.txt file, and came to know that one famous site robots.txt says
What it means ?
Stopping search engines to crawl http or https sites means all the pages?
P.S. That website have many pages ranked on google top pages.