Disallow Widget Php Directory in robots.txt
Hi everybody.
I have a weather forecast site that offers weather widgets to other sites.
The widget is an iframe that pulls data from, let's say, mysite.com/widget/yourforecast.php
Search engines follow the link and land on yourforecast.php page (which is not cached) causing not cached requests and fatiguing the server.
Can I disallow /widget/ from robots.txt without this penalizing me?
Doesn't Google like robots.txt that inhibit access to scripts?
Thank you
I have a weather forecast site that offers weather widgets to other sites.
The widget is an iframe that pulls data from, let's say, mysite.com/widget/yourforecast.php
Search engines follow the link and land on yourforecast.php page (which is not cached) causing not cached requests and fatiguing the server.
Can I disallow /widget/ from robots.txt without this penalizing me?
Doesn't Google like robots.txt that inhibit access to scripts?
Thank you
