Disallow Widget Php Directory in robots.txt

by Hims
0 replies
Hi everybody.
I have a weather forecast site that offers weather widgets to other sites.

The widget is an iframe that pulls data from, let's say, mysite.com/widget/yourforecast.php

Search engines follow the link and land on yourforecast.php page (which is not cached) causing not cached requests and fatiguing the server.
Can I disallow /widget/ from robots.txt without this penalizing me?
Doesn't Google like robots.txt that inhibit access to scripts?

Thank you
#directory #disallow #php #robotstxt #widget
Avatar of Unregistered

Trending Topics