How to use competitor data to improve my website

22 replies
UPDATED:

The Long, Tedious Journey is Over!
(Jump to the end for the solution I found if you're in a hurry!)

I set out to scrape competitor metadata, webpage content, and a sample of backlinks and citations.

Why work with raw data instead of fancy analysis tools?

Simply put, while AI is powerful, only the human brain--whether it belongs to the business owner or marketing staff--can truly compare and assess how competitor data stacks up for SEO.

I always identify competitors manually by searching relevant keywords locally. Forget all those easy online tools--they just generate endless lists that still require manual review for relevance. Either way, there's no shortcut to physically verifying the results.

Once I have the final competitor URLs, I extract a few backlinks, citations, and mentions for each one. The next step involves scraping the best 5-10 relevant competitors' web pages and filtering the most valuable backlinks or mentions. I focus on critical data points such as titles, the context of mentions, backlinks with anchor text, and reviews (if relevant)--all essential SEO ranking factors based on Google's algorithm.

The metadata, content, and referencing sites' insights create, in my experience, the foundation of a strong SEO benchmark. Despite being time-consuming, nothing beats doing this research yourself--or at least automating it smartly. That said, this is just one aspect of SEO, and there's much more to the process, which I won't cover in this post.

Automation or Manual Work? The Search for a System

Since this process requires hours of work for every round, I wanted to either automate it myself or find a system that could do it efficiently. Time is money, so I was willing to pay a reasonable price for a reliable solution.

Thanks to @Marx Vergel Melencio insights (big shout-out to you, my friend!), I initially tried coding a solution with a programmer buddy. While it was partially doable, I quickly ran into API costs, maintenance issues, hosting challenges, and extra coding resource needs--sending me right back to square one.

Over the past couple of months, I have tested more than 20 different tools, all of which failed to provide the competitor data I consider essential. I even tested the API recommended by @Marx Vergel Melencio, which scrapes almost anything on the web, but it lacked automation and wasn't SEO-oriented. The paid version also brought me back to the same roadblocks.

The Solution: Webcarrots.com

After much trial and error when it comes to how to find competitors the perfect way, I tested webcarrots this past week, and it automates about 90% of what I need.

The pros: This system scrapes metadata and separately extracts H1 headings, full webpage content, and structures it into an online report. The report includes another tab with top-ranking SERP sites based on live searches (which I verified). This feature is valuable for tracking historical data when monitoring web pages over time.

The cons: keep in mind that webcarrots.com does not provide comparison tools or analytics--it is strictly a raw data machine. They do mention upcoming features, but if you're looking for fancy graphic reports, this isn't the tool for you.

Is It worth the cost?

I'm 100% satisfied with the less-than-100% product of my dreams. It's cost-effective and automates a significant portion of my competitor research once I have my target list, saves me at least 1-2 hours of manual work.

Important Caveats!

Industry Restrictions: webcarrtos restricts scraping URL's of certain industries, such as government, financial, and news sites. When I tested this, it blocked submissions for these URLs.
It seems to have some rate limits resulting in temporary Blocks: When I ran multiple news site searches just to see what will happen, I encountered temporary form field block after a few tries. TIP: Clearing cookies and using the emailed login link resolved this issue, so I'm sharing this tweak for anyone facing the same problem.

Wrapping Up

Until my next journey
#competitor #data #improve #website
Avatar of Unregistered
  • Profile picture of the author Monetize
    Originally Posted by Mel Dart View Post

    I found a cheaper alternative to semrush etc and (not naming it due to rules) that provided competitors insights. The report per competitor includes the full content of competitor web pages in plain text, metadata (titles, etc.), and 5 sources where they're mentioned in search engines (socials, directories, etc.), with more details like titles and excerpts from each site.

    I know I can watch youtubes for a week, but I'm looking for experienced guidance to give me a 101 or starter tips on using this data to improve my site's content, keywords, and titles to outshine the competition.

    Whether you should follow your competition depends
    on a number of factors such as the industry, who they
    are, and whether you are anywhere near them as far
    as being an actual competitor goes.

    If you are a small fry, then you probably do not need to
    concern yourself about what a whale is doing.

    You need to learn about your online business, just as
    you would with any business. There are no shortcuts
    just because it is internet related.

    If you know you can watch YouTubes for a week, then
    that is what you should do. Maybe even years. I have
    been MMO for 20+ years and I'm still learning.

    A week is a blink of an eye. If you can't spend a few
    days learning about some aspect of your business,
    you are obviously not serious about it.
    {{ DiscussionBoard.errors[11816035].message }}
    • Profile picture of the author Mel Dart
      Firstly, I really appreciate your reply and the valuable tips. You're absolutely right about the importance of learning. My concern was that semrush might be overkill for my needs, so I didn't want to invest time in learning it only to not use it.

      If you know of a more affordable alternative that focuses specifically on competitor insights--such as extracting meta data and web page content to improve my content strategy and discover proven keyword ideas, I'd be happy to invest time in learning how to use it effectively.

      I've been talking to others, and everyone agrees that having competitor data from top SERP sites is essential. Of course, backlinks and citations play a role too, but analyzing their meta data, content, and top citations/backlinks is key to understanding why they rank above me.

      The ideal system for me would allow me to gather this data from 7-8 competitor sites--meta tags, content, and their strongest backlinks/citations, so I can analyze it thoroughly and refine my strategy.

      Right now, I'm doing this manually, and it's incredibly tedious and time-consuming especially since our competitors change prices and offers every month. I really need an automated solution to keep up with these changes efficiently!
      {{ DiscussionBoard.errors[11816299].message }}
      • Profile picture of the author Monetize
        Originally Posted by Mel Dart View Post

        Firstly, I really appreciate your reply and the valuable tips. You're absolutely right about the importance of learning. My concern was that semrush might be overkill for my needs, so I didn't want to invest time in learning it only to not use it.

        If you know of a more affordable alternative that focuses specifically on competitor insights--such as extracting meta data and web page content to improve my content strategy and discover proven keyword ideas, I'd be happy to invest time in learning how to use it effectively.

        I've been talking to others, and everyone agrees that having competitor data from top SERP sites is essential. Of course, backlinks and citations play a role too, but analyzing their meta data, content, and top citations/backlinks is key to understanding why they rank above me.

        The ideal system for me would allow me to gather this data from 7-8 competitor sites--meta tags, content, and their strongest backlinks/citations, so I can analyze it thoroughly and refine my strategy.

        Right now, I'm doing this manually, and it's incredibly tedious and time-consuming especially since our competitors change prices and offers every month. I really need an automated solution to keep up with these changes efficiently!

        You can do whatever you want but keeping track of eight
        "competitors" is a bit much.

        I don't understand how you operate your business when
        you are keeping up with that many other businesses and
        their websites.

        I recommend you narrow this down to one or two, pick
        the most successful ones, the rest do not matter.

        There's nothing much to learn about Semrush, I use it for
        keyword research, if there is anything else that it's useful
        for, IDK what it is and I don't have time for it.

        Maybe someone else will come along and enlighten me.

        Get SpyFu, A.I., and focus on your own business instead
        of what other people are doing.

        As far as costs of subscriptions, you don't have to sign up
        forever. Sign up for a month or two, do whatever, cancel
        your subscription, and move on.

        Why some people act like it's the end of the world to spend
        $30 on their business is mind boggling.
        {{ DiscussionBoard.errors[11816367].message }}
        • Profile picture of the author Mel Dart
          You're absolutely right--there's no need to stress over $30 a month, especially since advanced tools like Semrush can easily cost hundreds monthly, which can be overkill.

          I'm dedicating this weekend to finding a solution to track 5-10 competitors (including their homepages, metadata, and titles) and am willing to pay up to $50 a month to save those hours of manual work.
          {{ DiscussionBoard.errors[11816428].message }}
      • @Mel Dart,

        A moderator might soon transfer your thread to the Search Engine Optimization forum ...

        Anticipating that's likely to happen, let's start talking about SEO, veering from general business dev core that I think you and Monetize were talking about earlier.

        Yes. If they're the top ranking Google results in my target geos and target devices for my target exact match and longtail search terms, then it's a good idea to understand why Google's serving 'em over everybody else.

        For me, my Google SE competitor analytics strategy's founded on Moz's DA (domain authority) and PA (page authority) algorithm.
        SEMRush's AS (authority score) algo just has a small influence over my strategy (I'd say ~10%).
        And if AR (Ahrefs Rank) has any influence over my strategy -- Mostly, that's only if I'm handling aged websites with decent DA and lots of content pages with decent PA to start with.

        So In any case, I target search terms well within the punching weight of the domain I'm handling.
        Doing that helps me focus on building DA first, then PA gradually, and I'll also be able to efficiently scale up to higher competitiveness levels.

        Yes, I analyze the metadata and content of top ranking pages whose domains have DA and PA surpassable by the current state of the domain I'm handling.
        But lately (2020 onwards), when I'm handling a brand new domain and website with small to no content -- I focus on genAI-infused programmatic SEO first.

        This is to build DA through domain knowledge base-type content (been getting good results for exact match and question-type search terms in competitive B2B and B2C niches across high ecommerce consumership geos).

        And once my DA and the PA of some pages improve a bit, I continue publishing the knowledge base stuff, while putting heavier focus on publishing other types of content, like for example news stories, guides and tutorials, opinionated critique, articles and announcements, among others.
        I publish this other stuff as short and longform posts for longtail search terms (while still using voice query-type and question search terms).
        And I use genAI-infused programmatic SEO for that as well.

        Plus, since then, I "outsource" Google SE competitor analytics to a capable genAI model (fairly recently chain-of-thought ones) during pSEO content post generation runtime.
        Signature
        Need Custom Programmatic SEO or GenAI Engineering Work Done? Drop Me an Email HERE ...
        • Chief Machine Learning Engineer @ ARIA Research (Sydney, AU)
        • Lead GenAI SEO Campaign Engineer @ Kiteworks, Inc. (SF, US)
        {{ DiscussionBoard.errors[11816372].message }}
        • Originally Posted by Marx Vergel Melencio View Post

          A moderator might soon transfer your thread to the Search Engine Optimization forum ...

          Anticipating that's likely to happen, let's start talking about SEO, veering from general business dev core that I think you and Monetize were talking about earlier.

          Yes. If they're the top ranking Google results in my target geos and target devices for my target exact match and longtail search terms, then it's a good idea to understand why Google's serving 'em over everybody else.

          For me, my Google SE competitor analytics strategy's founded on Moz's DA (domain authority) and PA (page authority) algorithm.
          SEMRush's AR (authority score) algo just has a small influence over my strategy (I'd say ~10%).
          And if AR (Ahrefs Rank) has any effect over my strategy -- Only if I'm handling aged websites with decent DA and lots of content pages with decent PA to start with.

          So In any case, I target search terms well within the punching weight of the domain I'm handling.
          Doing that helps me focus on building DA first, then PA gradually, and I'll also be able to efficiently scale up to higher competitiveness levels.

          Yes, I analyze the metadata and content of top ranking pages whose domains have DA and PA surpassable by the current state of the domain I'm handling.

          But lately (2020 onwards), when I'm handling a brand new domain and website with small to no content -- I focus on genAI-infused programmatic SEO first.

          This is to build DA through domain knowledge base-type content (been getting good results for exact match and question-type search terms in competitive B2B and B2C niches across high ecommerce consumership geos).

          And once my DA and the PA of some pages improve a bit, I continue publishing the knowledge base stuff, while putting heavier focus on publishing other types of content, like for example news stories, guides and tutorials, opinionated critique, articles and announcements, among others.
          I publish this other stuff as short and longform posts for longtail search terms (while still using voice query-type and question search terms).
          And I use genAI-infused programmatic SEO for that as well.

          Plus, since then, I "outsource" Google SE competitor analytics to a capable genAI model (fairly recently chain-of-thought ones) during pSEO content post generation runtime.
          Saw the synopsis ovah on GALS4CANDOR.com!

          Ain't seecrits such a beeitch!
          Signature

          Lightin' fuses is for blowin' stuff togethah.

          {{ DiscussionBoard.errors[11816379].message }}
  • Profile picture of the author catexotica
    [DELETED]
    {{ DiscussionBoard.errors[11816389].message }}
    • Profile picture of the author Mel Dart
      Thanks so much for sharing these strategies, I agree they're the building blocks,however, I'm looking to keep track of competitors' new offers, pricing changes, and similar updates on a monthly basis, which standard SEO tools like semrush, ahref etc don't fully cover.

      That's why I'm dedicating this weekend to finding a more focused competition tracking solution, preferably something automated. If you have any recommendations, I'd really appreciate your input!

      Have a great weekend!
      {{ DiscussionBoard.errors[11816430].message }}
      • @Mel Dart,

        Originally Posted by Mel Dart View Post

        [SNIP]
        I'm looking to keep track of competitors' new offers, pricing changes, and similar updates on a monthly basis
        preferably something automated
        If you have any recommendations, I'd really appreciate your input!
        [/SNIP]
        Just use OpenAI's dev API for their gpt-4o-mini model and SERPAPI.
        Here's the programmatic workflow that you can write as a Python script:

        • Pass your list of target exact match and longtail keywords to the LLM, prompting it to convert them into buyer keywords and then to full search strings with suitable Google search operators, in order to get top ranking ecommerce store results.

        =>

        • Give the LLM a function to use that data to perform autonomous Google searches through SERPAPI (don't forget to set SERPAPI params to your target geos and devices).

        =>

        • Auto-extract clean content of each page through libraries like requests and beautifulsoup in Python or premium API services like Extractor API.

        =>

        • Hand over extracted clean text to the LLM, prompting it to generate a response in JSON format for the keys and the values that you want (think of this as a spreadsheet where "keys" = "column headers" and "values" = data in cells under a column header).


        =>

        • Process the LLM's response and store it in a MongoDb and/or a local spreadsheet. Put it unto Google Sheets if you want.

        ===
        If you improve how to communicate the workflow above to an LLM right now -- I recommend using ChatGPT's O1 model (there are bundled free usage credits with weekly limits in ChatGPT Plus accounts) -- ChatGPT would be able to write the full Python code for you. And if you prompt it, it'll also guide you step-by-step on how to setup Python and how to run the script in your PC or Mac.
        Signature
        Need Custom Programmatic SEO or GenAI Engineering Work Done? Drop Me an Email HERE ...
        • Chief Machine Learning Engineer @ ARIA Research (Sydney, AU)
        • Lead GenAI SEO Campaign Engineer @ Kiteworks, Inc. (SF, US)
        {{ DiscussionBoard.errors[11816435].message }}
        • Profile picture of the author Mel Dart
          Hey Marx, you're a genius! I checked out your concept with a programmer friend, he also looked into serpapi and said it's doable if we set up a form to handle query parameters. His main concerns are the monthly costs and the automation part, which will need more extensive programming and some debugging until we get it right.

          We're dedicating all Sunday to this, and I'll update here if I can refine this focused concept into an automated process + work out the costs for serpapi which I saw the min is 75 a month and GPT API (after the free prompts) on 4 it is reasonable but the 1o can accumulate exponentially. I really think it could benefit a lot of people here if we nail it.

          Your info also helped me narrow my online search, I found two promising websites and already reached out to their support teams. They seem cheap, but let's see if they actually deliver without surprise fees, unlike good old semrush and others when you want the real good stuff.

          Thanks again and enjoy the rest of your weekend!
          {{ DiscussionBoard.errors[11816503].message }}
          • @Mel Dart,

            Originally Posted by Mel Dart View Post

            Hey Marx, you're a genius! I checked out your concept with a programmer friend, he also looked into serpapi and said it's doable if we set up a form to handle query parameters. His main concerns are the monthly costs and the automation part, which will need more extensive programming and some debugging until we get it right.

            We're dedicating all Sunday to this, and I'll update here if I can refine this focused concept into an automated process + work out the costs for serpapi which I saw the min is 75 a month and GPT API (after the free prompts) on 4 it is reasonable but the 1o can accumulate exponentially. I really think it could benefit a lot of people here if we nail it.

            Your info also helped me narrow my online search, I found two promising websites and already reached out to their support teams. They seem cheap, but let's see if they actually deliver without surprise fees, unlike good old semrush and others when you want the real good stuff.

            Thanks again and enjoy the rest of your weekend!

            Glad to help.
            • SERPAPI has a free tier subscription with 2000 monthly searches -- This is more than enough for your use case, judging from how you explained here what you need; and
            • You won't need to use the o1, O1/o3-mini and o3-mini-high models of OpenAI -- I recommended this family of OpenAI models only for code generation, i.e. If you want to generate the code on your own by simply feeding it with the process workflow description in my previous reply here. However:

            ** For the use case of your actual script during runtime -- gpt-4o-mini is ideal;
            ** OpenAI has a pay-as-you-go subscription. For the gpt-4o-mini model, you can ingest (analyze) CONTENT totalling to 600K++ words, collectively across multiple iterations (one iteration step maxes out to 85K++ input words), for $0.15 (that's total cost for analyzing 600K++ words of structured and unstructured data, which you'll extract from top ranking Google results pages matching your parameters in your SERPAPI calls, and for extraction, you can do this through requests and beautifulsoup or ExtractorAPI); and
            ** Because you just need spreadsheet-type data as output of the data analytics workflow -- This won't cost a lot, as this is just a few hundred-low thousand words per entry. And, output generation using gpt-4o-mini costs $0.60 for 600K++ words of output (collectively, since gpt-4o-mini maxes out to just 13K++ generated words per turn).

            [EDIT: I suggest not limiting yourself to just organic Google results for analytics -- Instead, you can also use SERPAPI's Google Ads Transparency Center API to include matching Google ads in your SERPAPI calls, considering that many if not most of your top competitors are likely advertising in GOOGLE AS WELL. And, here's more info about this: https://serpapi.com/google-ads-transparency-center-api ]

            Cheers!
            Signature
            Need Custom Programmatic SEO or GenAI Engineering Work Done? Drop Me an Email HERE ...
            • Chief Machine Learning Engineer @ ARIA Research (Sydney, AU)
            • Lead GenAI SEO Campaign Engineer @ Kiteworks, Inc. (SF, US)
            {{ DiscussionBoard.errors[11816540].message }}
            • Profile picture of the author Mel Dart
              We're deep into this now. A couple of things. serpapi's free plan only allows 100 searches, and we've already exhausted that with all the testing. The next option is 75 a month for 5000 searches.

              For openai, you nailed it. But one caveat, the input part of the prompt requires detailed logic explanation and dynamic data as a string. So far, the input cost is around 0.41, and the output we're using further adds about 0.02. If I really push it like today, it could add up to 0.50, which means just a few bucks a month, not a big deal.

              Bottom line, it's a workable solution. But if I rely only on the free serpapi, that 75 a month is steep. Not sure if I'll run out of searches before I get to my most important research.

              Another thing, neither openai nor serpapi is scraping the data. This is moving into selenium development, which makes it more complex. My developer says it's doable but requires professional programming to automate fully and generate the report I want. Plus, there's the AWS server cost.

              Again, I really appreciate your help. This has been a great learning experience. I've already received a response from one website and will test their system. It's just 8 a month, no biggie to try. If it's anywhere close to what I need, I'll update everyone here.

              Many thanks, my friend. You really helped a lot. I feel totally educated now and much more focused on what I want.
              {{ DiscussionBoard.errors[11816555].message }}
              • @Mel Dart,

                Originally Posted by Mel Dart View Post

                [SNIP]serpapi's free plan only allows 100 searches, and we've already exhausted that with all the testing. The next option is 75 a month for 5000 searches.[/SNIP]
                Hmmm ... Might've got that free subscription with 2K searches (I just checked, and it's active) from aChristmas promo or something.

                Originally Posted by Mel Dart View Post

                [SNIP]the input part of the prompt requires detailed logic explanation and dynamic data as a string. So far, the input cost is around 0.41, and the output we're using further adds about 0.02. If I really push it like today, it could add up to 0.50, which means just a few bucks a month, not a big deal.[/SNIP]
                Hmmm ... That's strange.

                It didn't cost me too much in terms of gpt-4o-mini dev API pay-as-you-go credits to complete a fully automated half-a-million-word pSEO campaign through my private custom WP plugin for my finance and investment niche WP site.

                I completed that pSEO campaign just last week.
                Per 2-way API call (ingestion and generation) -- Running this pSEO campaign didn't cost me a lot to use gpt-4o-mini.

                And cost was and still is a lot less than $0.01 per word.
                For that project, I send explanations about required logic and dynamic data from sanitized text extracted from filing data of matching form types in United States SEC database (requires analytics of multiple form types to determine match likelihood, classify, generate and format unstructured data to structured JSON data in the target format) and resulting pages of Google Web searches (I set this to top 3 for "relevance"), and different set of search terms in Extractor API's nnews search API (also set to top 3), in which both also output unstructured data, a fairly good reason to use AI analytics.

                This includes extracted research data, full illustrative HTML-coded examples and instructions for technical structure, metadata style and format, FAQs schema markup from dynamically supplied SEO data (questions in Google's PAA and PASF), header outline, content with links, niche topic, depth, voice, tone and style, comprehensive details about search trends in target niche and niche audience's content viewing preferences.

                And, completion of a pSEO content post requires multiple iterations, like:

                Analyze comprehensive details supplied for target niche audience and geos plus SEO data harvested from Google Trends and Google Ads API ...

                =>
                Convert understanding to suitable search terms in Google and required arguments for hardcoded data API sources (U.S. SEC filing data and Extractor API's news search API for this project) ...

                =>
                Analyze research data extracted from Google Web search results pages and hardcoded API data sources, then generate content ...

                =>
                And so on, such as automated post cleanup, comparing if content is duplicate or highly similar to previously generated posts scheduled for future publication in WP, etc.

                Originally Posted by Mel Dart View Post

                [SNIP]Another thing, neither openai nor serpapi is scraping the data. This is moving into selenium development, which makes it more complex. My developer says it's doable but requires professional programming to automate fully and generate the report I want.[/SNIP]
                In my earlier posts here, I mentioned using requests to extract the content from the URLs that will be supplied by SERPAPI as the results of your autonomous Web searches, and then use beautifulsoup to sanitize the text.
                Those are Python libraries that can be used to not cost too much time to code a script for your use case. An hour tops for senior Python engineers (8++ years). 3 max for junior (~3 years).
                And, this method is way, way more stable than Selenium scripts in either Python or Javascript.

                Also, I mentioned Extractor API [ https://extractorapi.com/ ].
                I use this to significantly decrease blocked page errors through Javascript-powered renderings, security tokenizations and others, instead of doing these things on my own.
                They offer a pay-as-you-go account with I think 100 searches per month free.

                Originally Posted by Mel Dart View Post

                [SNIP]Plus, there's the AWS server cost.[/SNIP]
                If you install Python 3.11 in your machine (Windows or Mac or Linux) -- You can run this script whenever you need it. You don't need to host this. It's unnecessary.
                In case you aren't familiar with Python, you can run your script through the CMD terminal, similar to how we ran .bat and .exe files in CMD during the good old MS DOS days.

                Originally Posted by Mel Dart View Post

                [SNIP]Again, I really appreciate your help. This has been a great learning experience. I've already received a response from one website and will test their system. It's just 8 a month, no biggie to try. If it's anywhere close to what I need, I'll update everyone here.

                Many thanks, my friend. You really helped a lot. I feel totally educated now and much more focused on what I want.[/SNIP]
                Glat to help!
                Others might also find some value in this thread, which seems to be a mix of quick glance core business fundamentals, competitor analytics task automation, and programmatic SEO.
                Signature
                Need Custom Programmatic SEO or GenAI Engineering Work Done? Drop Me an Email HERE ...
                • Chief Machine Learning Engineer @ ARIA Research (Sydney, AU)
                • Lead GenAI SEO Campaign Engineer @ Kiteworks, Inc. (SF, US)
                {{ DiscussionBoard.errors[11816563].message }}
                • Profile picture of the author Mel Dart
                  Thanks again!!

                  You're spot on with the goal here! I'm currently testing https://extractorapi.com/ as one of two options for my needs. If it performs well, I'll likely go with the $33/month package. The only thing I'm not 100% clear on yet is the custom automation, but as you mentioned, I might handle that with Python.

                  Since I'm collaborating with a colleague, this needs to be hosted, so I'm looking at around $50/month total. The good part - ExtractorAPI also covers the AI ID I need, so I won't need an extra OpenAI account.

                  I've sent out some requests for freelance Python work, seems like toptier devs value their time pretty high these days. That's fine, though, if we get exactly what we need, there's no way around it. At least it's a one-time cost, though of course, we'll need someone to handle bugs and upgrades as we go.

                  Fingers crossed for the other site too. From what they offer, it looks like they can do it all, but past experience tells me there can sometimes be a gap between promise and delivery. Let's see how it plays out.

                  As the week starts, I plan to check out the other site tonight or by tomorrow at the latest, I'll provide all the details once I do.
                  {{ DiscussionBoard.errors[11816599].message }}
  • Profile picture of the author mindtricks18
    Yes, it will be great support if I can find SEMrush alternative at cheap price
    {{ DiscussionBoard.errors[11816450].message }}
    • Profile picture of the author Mel Dart
      Hi, the solution I'm after is to automate the tracking of competition-specific web pages to extract their metadata like title, description, and content for the tracked page, along with at least three backlinks or citations from sites that mention them and appear on the first page when I look them up.

      This is something I currently do manually, which is time-consuming, but there's no way around it if you need to track competition, monitor price changes, added offers, and analyze all of that to make informed decisions.

      As for getting the competition list, I found that neither semrush nor ahrefs accurately identify my exact competitors. They pull data that's sometimes outdated and incomplete. While I know my main competitors, a manual check is still the best way to verify if any new players have entered the market, especially since competition depends on my physical location. There are huge discrepancies between my own search results and what semrush provides when simulating my location. That part is easy, the real challenge is tracking and extracting the data.

      I've tested many concepts, and I know for sure that staying up to date with competitor content on SERP-ranked pages for my target keywords provides the best insights for ad strategies and content efforts.

      semrush, in a way, makes you lazy by offering all these content and keyword ideas theoretically. They have great tools, but for me, this simple concept covers everything with precision, focused on the exact competitors. I then handle keyword research and content optimization using a mix of free tools, ChatGPT, and my own judgment, rather than relying on a non-real-time system.
      {{ DiscussionBoard.errors[11816504].message }}
    • Profile picture of the author Mel Dart
      I hope I'm not chasing something that doesn't exist. Of course, the monthly cost is also a key factor. If I find it, I'll be happy to share.
      {{ DiscussionBoard.errors[11816505].message }}
  • Profile picture of the author 00CashBack
    I have tried many alternative to Semrush, but now, i'm back to Semrush. It's the best i think. Expansive, but using it give my website more visibility day after day.
    {{ DiscussionBoard.errors[11816480].message }}
  • Profile picture of the author dexcowork
    Analyzing competitors is such an effective strategy to identify opportunities and gaps. The suggestions on improving SEO and user experience are spot-on. Thanks for sharing these valuable insights
    {{ DiscussionBoard.errors[11816603].message }}
  • Profile picture of the author Mel Dart
    Just a quick note that I have updated the main post to incorporate information addressing multiple replies.
    {{ DiscussionBoard.errors[11816847].message }}
  • Jus' wanna check, babes ...

    once'n yr done with yr various metrics ...

    whatchyoo actschwlly SAYIN' to nowan lookin' in on yr ULTRA SPESH hideyhole?

    Gravity without splat is jus' air.

    tbh plenny websites fail bcs they stick so much schwango buttween audience desiah an' playah evokesy.

    Like the Dipsy Schwipsy Swivel Gals said back the las' time we had Twennies breakout stuffs packin' maxo flap:

    "To hell with our outfits -- this is us!"
    Signature

    Lightin' fuses is for blowin' stuff togethah.

    {{ DiscussionBoard.errors[11816857].message }}
  • i use similar web its great for data, and many features on it, free trail included too
    Signature

    Lets build a online business by giving value and learning how to build a email list
    https://givevaluefirst.systeme.io/givevalueonwarriorf

    {{ DiscussionBoard.errors[11816995].message }}
    • Profile picture of the author Mel Dart
      Yes I tried it, you're right it is great for comparisons and visuals, but it is not scraping the data like webcarrots, so I switched.

      Just to clarify, it's not a full alternative, but for the price and the competitor data you get, WebCarrots is doing the job. I'm using it, new to it, but so far, it seems like a solid option.
      {{ DiscussionBoard.errors[11817001].message }}
  • Profile picture of the author Bimble Exc
    Banned
    Thanks so much for sharing these strategies, I agree they're the building blocks,however, I'm looking to keep track of competitors' new offers, pricing changes, and similar updates on a monthly basis, which standard SEO tools like semrush, ahref etc don't fully cover.

    That's why I'm dedicating this weekend to finding a more focused competition tracking solution, preferably something automated. If you have any recommendations, I'd really appreciate your input!

    Have a great weekend!
    {{ DiscussionBoard.errors[11817022].message }}
Avatar of Unregistered

Trending Topics