7 {{ upvoteCount | shortNum }}
7 {{ upvoteCount | shortNum }}

Bots scraping data from your website

by WF- Enzo Administrator
6 replies
Do you mind bots scraping data from your website?

Well as you know, data from websites is publicly-available information so it's always up fro grabs for any visitor. Some people even think that if you don't want information to be collected from your website then don't put it on your website in the first place.

On the other hand, bot scraping is an automated process of information gathering - more efficient can humans ever hope to be. And that data can be used to make the online experience better.

Chime in.
#bots #data #scraping #website
Avatar of Unregistered
  • Profile picture of the author digitalstar
    Normally we won't block bots from scraping data from our websites. But there is the possibility of blocking them using robots.txt
    Signature
    Product Reviews | Service Reviews | Freebies | Giveaways | Super Deals
    {{ DiscussionBoard.errors[11683464].message }}
  • Profile picture of the author Profit Traveler
    Banned
    Facebook article on scraping...on the "Meta" website.

    Each article also linked to another at the bottom on same topic.

    https://about.fb.com/news/2021/04/ho...mbat-scraping/

    https://about.fb.com/news/2021/04/fa...facebook-data/
    Signature
    MAKE MONEY DOING WHAT YOU ARE ALREADY DOING...

    SOCIAL MEDIA MONETIZATION ON STEROIDS

    {{ DiscussionBoard.errors[11683578].message }}
  • Profile picture of the author praetserge
    I was actually considering blocking so bots like ahrefs, semrush, etc.
    All they do is... help your competitors and use your bandwidth.
    But it's better to block them through .htaccess file.
    Signature
    QUORA Answers Which Stick. Boost your Website by 250 Word 2 Unique QUORA Answers with contextual link for $1
    {{ DiscussionBoard.errors[11692186].message }}
  • Profile picture of the author CyberSEO
    Originally Posted by WF- Enzo View Post

    On the other hand, bot scraping is an automated process of information gathering - more efficient can humans ever hope to be. And that data can be used to make the online experience better.
    Do you know how Google News, Yahoo News, MSN News and other content aggregating platforms work? these are just bots that scrape the Internet for the news. You can call them "autoblogs", because this is that they really are.



    So there is nothing wrong with bots scrapping your sites. Just make sure you have links and copyright notes in your posts, so the readers know about your resources. Lots of my own customers run huge networks of sites like that. If they do it right, they are absolutely ok with that.
    Signature
    CyberSEO Pro - the almighty content syndicator for WordPress with a wide range of cutting edge AI technologies for SEO, such as OpenAI GPT-3, DALL·E 2, DeepL, Article Forge and others. Show me a better solution and I'll pay you $1000 in reward!
    {{ DiscussionBoard.errors[11692498].message }}
  • Profile picture of the author Jonathan S
    Most search engines are using bots to scrape your website and increase your visibility and rankings. You can customize the robot.txt to tell the bots what to scrape and not to scrape.
    Signature
    {{ DiscussionBoard.errors[11692633].message }}
  • Profile picture of the author coolidge
    Originally Posted by WF- Enzo View Post

    Do you mind bots scraping data from your website?

    Well as you know, data from websites is publicly-available information so it's always up fro grabs for any visitor. Some people even think that if you don't want information to be collected from your website then don't put it on your website in the first place.

    On the other hand, bot scraping is an automated process of information gathering - more efficient can humans ever hope to be. And that data can be used to make the online experience better.

    Chime in.
    Bots filling out forms on the website is the most annoying thing.
    Signature

    ----------------------------------------------------------------------------------------
    *** UNLIMITED TRAFFIC ON DEMAND! *** TARGET BUYERS = GET SALES!!
    ----------------------------------------------------------------------------------------

    {{ DiscussionBoard.errors[11692740].message }}
Avatar of Unregistered

Trending Topics