Screaming Frog & not factoring in URLs

4 replies
  • SEO
  • |
Hello there,

When I enter the homepage URL of a website I'm working on into screaming frog, it crawls and provides information on 37 different URLs.

However, I notice the screaming frog report doesn't crawl all the URLs linked to the sub domain. This is where it gets confusing for me...

I can use a Google Search to find the URLs that the screaming frog report ignores and can then manually enter these URLs into screaming frog to get information i.e. Header tags, meta data etc.

However, these URLs that I can find on Google DON'T appear to be accessible through the main website, so I assume that's why screaming frog doesn't automatically crawl these URLs? Although when I manually enter these URLs that are ignored, screaming frog provides information on a range of different pages that ARE accessible on the main website.

If I manually enter a URL that IS NOT accessible on the main website, then why does screaming frog provide multiple URLs that CAN be found on the website? It just seems contradictory...

Sorry if this sounds confusing. I'm a relative Newbie to SEO.
#factoring #frog #screaming #urls
  • Profile picture of the author yukon
    Banned
    SEO Spider Configuration | Screaming Frog

    Note: this only works If you have a payed license.

    Crawl all subdomains

    By default the SEO Spider will only crawl the subdomain you crawl from and treat all other subdomains encountered as external sites. To crawl all subdomains of a root domain, use this option.
    {{ DiscussionBoard.errors[10710326].message }}
    • Profile picture of the author SEO JOEL
      Originally Posted by yukon View Post

      SEO Spider Configuration | Screaming Frog

      Note: this only works If you have a payed license.
      Thanks for your reply Yukon.

      What exactly would the 'other subdomains' constitute? The URLs I search for manually on Google share the same sub domain address, as pages accessible on the website I'm working on i.e. 'www.aboutus.com'
      {{ DiscussionBoard.errors[10710690].message }}
      • Profile picture of the author yukon
        Banned
        Originally Posted by SEO JOEL View Post

        Thanks for your reply Yukon.

        What exactly would the 'other subdomains' constitute? The URLs I search for manually on Google share the same sub domain address, as pages accessible on the website I'm working on i.e. 'www.aboutus.com'
        In Screaming Frog go to the top/left menu and select:
        • Configuration > Spider

        That will show you a list of options (including crawl all sub-domains), some will be grayed out unless you buy a license for the software.

        The other sub-domains would be any sub-domain that you aren't searching for when you click the Start button.

        Example, If you're searching for cars.about.com Screaming Frog will ignore links on music.about.com unless you have a payed license because it's not the same sub-domain that you originally searched (ex: cars.about.com).
        {{ DiscussionBoard.errors[10710825].message }}
  • Profile picture of the author SEO JOEL
    Thanks for reply and your patients once again Yukon.
    {{ DiscussionBoard.errors[10711344].message }}

Trending Topics