
How to use competitor data to improve my website
The Long, Tedious Journey is Over!
(Jump to the end for the solution I found if you're in a hurry!)
I set out to scrape competitor metadata, webpage content, and a sample of backlinks and citations.
Why work with raw data instead of fancy analysis tools?
Simply put, while AI is powerful, only the human brain--whether it belongs to the business owner or marketing staff--can truly compare and assess how competitor data stacks up for SEO.
I always identify competitors manually by searching relevant keywords locally. Forget all those easy online tools--they just generate endless lists that still require manual review for relevance. Either way, there's no shortcut to physically verifying the results.
Once I have the final competitor URLs, I extract a few backlinks, citations, and mentions for each one. The next step involves scraping the best 5-10 relevant competitors' web pages and filtering the most valuable backlinks or mentions. I focus on critical data points such as titles, the context of mentions, backlinks with anchor text, and reviews (if relevant)--all essential SEO ranking factors based on Google's algorithm.
The metadata, content, and referencing sites' insights create, in my experience, the foundation of a strong SEO benchmark. Despite being time-consuming, nothing beats doing this research yourself--or at least automating it smartly. That said, this is just one aspect of SEO, and there's much more to the process, which I won't cover in this post.
Automation or Manual Work? The Search for a System
Since this process requires hours of work for every round, I wanted to either automate it myself or find a system that could do it efficiently. Time is money, so I was willing to pay a reasonable price for a reliable solution.
Thanks to @Marx Vergel Melencio insights (big shout-out to you, my friend!), I initially tried coding a solution with a programmer buddy. While it was partially doable, I quickly ran into API costs, maintenance issues, hosting challenges, and extra coding resource needs--sending me right back to square one.
Over the past couple of months, I have tested more than 20 different tools, all of which failed to provide the competitor data I consider essential. I even tested the API recommended by @Marx Vergel Melencio, which scrapes almost anything on the web, but it lacked automation and wasn't SEO-oriented. The paid version also brought me back to the same roadblocks.
The Solution: Webcarrots.com
After much trial and error when it comes to how to find competitors the perfect way, I tested webcarrots this past week, and it automates about 90% of what I need.
The pros: This system scrapes metadata and separately extracts H1 headings, full webpage content, and structures it into an online report. The report includes another tab with top-ranking SERP sites based on live searches (which I verified). This feature is valuable for tracking historical data when monitoring web pages over time.
The cons: keep in mind that webcarrots.com does not provide comparison tools or analytics--it is strictly a raw data machine. They do mention upcoming features, but if you're looking for fancy graphic reports, this isn't the tool for you.
Is It worth the cost?
I'm 100% satisfied with the less-than-100% product of my dreams. It's cost-effective and automates a significant portion of my competitor research once I have my target list, saves me at least 1-2 hours of manual work.
Important Caveats!
Industry Restrictions: webcarrtos restricts scraping URL's of certain industries, such as government, financial, and news sites. When I tested this, it blocked submissions for these URLs.
It seems to have some rate limits resulting in temporary Blocks: When I ran multiple news site searches just to see what will happen, I encountered temporary form field block after a few tries. TIP: Clearing cookies and using the emailed login link resolved this issue, so I'm sharing this tweak for anyone facing the same problem.
Wrapping Up
Until my next journey
⢠Chief Machine Learning Engineer @ ARIA Research (Sydney, AU)
⢠Lead GenAI SEO Campaign Engineer @ Kiteworks, Inc. (SF, US)
Lightin' fuses is for blowin' stuff togethah.
⢠Chief Machine Learning Engineer @ ARIA Research (Sydney, AU)
⢠Lead GenAI SEO Campaign Engineer @ Kiteworks, Inc. (SF, US)
⢠Chief Machine Learning Engineer @ ARIA Research (Sydney, AU)
⢠Lead GenAI SEO Campaign Engineer @ Kiteworks, Inc. (SF, US)
⢠Chief Machine Learning Engineer @ ARIA Research (Sydney, AU)
⢠Lead GenAI SEO Campaign Engineer @ Kiteworks, Inc. (SF, US)
DJI T50, Drone Cone
Lightin' fuses is for blowin' stuff togethah.
Lets build a online business by giving value and learning how to build a email list
https://givevaluefirst.systeme.io/givevalueonwarriorf