8 {{ upvoteCount | shortNum }}
8 {{ upvoteCount | shortNum }}

Bots scraping data from your website

by WF- Enzo Administrator
6 replies
Do you mind bots scraping data from your website?

Well as you know, data from websites is publicly-available information so it's always up fro grabs for any visitor. Some people even think that if you don't want information to be collected from your website then don't put it on your website in the first place.

On the other hand, bot scraping is an automated process of information gathering - more efficient can humans ever hope to be. And that data can be used to make the online experience better.

Chime in.
#bots #data #scraping #website
  • Profile picture of the author digitalstar
    Normally we won't block bots from scraping data from our websites. But there is the possibility of blocking them using robots.txt
    Signature
    Product Reviews | Service Reviews | Freebies | Giveaways | Super Deals
    {{ DiscussionBoard.errors[11683464].message }}
  • Profile picture of the author Profit Traveler
    Banned
    Facebook article on scraping...on the "Meta" website.

    Each article also linked to another at the bottom on same topic.

    https://about.fb.com/news/2021/04/ho...mbat-scraping/

    https://about.fb.com/news/2021/04/fa...facebook-data/
    {{ DiscussionBoard.errors[11683578].message }}
  • Profile picture of the author praetserge
    I was actually considering blocking so bots like ahrefs, semrush, etc.
    All they do is... help your competitors and use your bandwidth.
    But it's better to block them through .htaccess file.
    Signature
    FREE $5 coupon for SEO Clerks - start building links (guest posts, niche eidts, etc) to your site for FREE. DM me for your FREE coupon. Only for new customers.
    {{ DiscussionBoard.errors[11692186].message }}
  • Profile picture of the author CyberSEO
    Originally Posted by WF- Enzo View Post

    On the other hand, bot scraping is an automated process of information gathering - more efficient can humans ever hope to be. And that data can be used to make the online experience better.
    Do you know how Google News, Yahoo News, MSN News and other content aggregating platforms work? these are just bots that scrape the Internet for the news. You can call them "autoblogs", because this is that they really are.



    So there is nothing wrong with bots scrapping your sites. Just make sure you have links and copyright notes in your posts, so the readers know about your resources. Lots of my own customers run huge networks of sites like that. If they do it right, they are absolutely ok with that.
    Signature
    CyberSEO Pro - the ultimate all-in-one autoblogging WordPress plugin, powered by OpenAI GPT-4, Anthropic Claude, Google Gemini Pro, Midjourney, DALL-E 3 and Stable Diffusion XL
    {{ DiscussionBoard.errors[11692498].message }}
  • Profile picture of the author Jonathan S
    Most search engines are using bots to scrape your website and increase your visibility and rankings. You can customize the robot.txt to tell the bots what to scrape and not to scrape.
    Signature
    {{ DiscussionBoard.errors[11692633].message }}
  • Profile picture of the author coolidge
    Originally Posted by WF- Enzo View Post

    Do you mind bots scraping data from your website?

    Well as you know, data from websites is publicly-available information so it's always up fro grabs for any visitor. Some people even think that if you don't want information to be collected from your website then don't put it on your website in the first place.

    On the other hand, bot scraping is an automated process of information gathering - more efficient can humans ever hope to be. And that data can be used to make the online experience better.

    Chime in.
    Bots filling out forms on the website is the most annoying thing.
    Signature

    ----------------------------------------------------------------------------------------
    *** UNLIMITED TRAFFIC ON DEMAND! *** TARGET BUYERS = GET SALES!!
    ----------------------------------------------------------------------------------------

    {{ DiscussionBoard.errors[11692740].message }}

Trending Topics