Screaming Frog & not factoring in URLs
- SEO |
When I enter the homepage URL of a website I'm working on into screaming frog, it crawls and provides information on 37 different URLs.
However, I notice the screaming frog report doesn't crawl all the URLs linked to the sub domain. This is where it gets confusing for me...
I can use a Google Search to find the URLs that the screaming frog report ignores and can then manually enter these URLs into screaming frog to get information i.e. Header tags, meta data etc.
However, these URLs that I can find on Google DON'T appear to be accessible through the main website, so I assume that's why screaming frog doesn't automatically crawl these URLs? Although when I manually enter these URLs that are ignored, screaming frog provides information on a range of different pages that ARE accessible on the main website.
If I manually enter a URL that IS NOT accessible on the main website, then why does screaming frog provide multiple URLs that CAN be found on the website? It just seems contradictory...
Sorry if this sounds confusing. I'm a relative Newbie to SEO.
-
yukon Banned-
Thanks - 1 reply
{{ DiscussionBoard.errors[10710326].message }}-
SEO JOEL -
Thanks - 1 reply
{{ DiscussionBoard.errors[10710690].message }}-
yukon Banned-
Thanks
{{ DiscussionBoard.errors[10710825].message }} -
-
-
-
SEO JOEL -
Thanks
{{ DiscussionBoard.errors[10711344].message }} -