9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

Question about Google bot crawling

by dp40oz
8 replies
Theres something I don't quite get about how Googles bot crawls and indexes links. Lets say I wanted to get 1000 backlinks indexed. In theory I should be able to create a new post or page on one of my blogs that contains all 1000 links. Ping that post or bookmark. Now once Google picks up and indexes that post shouldn't it also crawl every link and now have them all indexed? This never seems to index many of the links. I would figure a bot would just index everything it crawled that way it would not be missing anything. I don't get how the post or page will get indexed and it just misses all the content. Where am I going wrong with this theory?
#bot #crawling #google #question
  • Profile picture of the author auradev
    They never seem to index all of the pages or links. I'm not sure why either. I don't think it has to do with them not being able to find the pages.
    {{ DiscussionBoard.errors[3287490].message }}
  • Profile picture of the author NeilC
    You need to be very careful with a links post or page, search engines generally don't like them these days and it could damage your rankings.

    The links page is usually seen as providing no real value and you need to make sure that the links are not "no-follow" links for the bots to be able to follow them.
    Signature

    Need a fresh start or help to take your business to the next level? Click here to find out more...

    {{ DiscussionBoard.errors[3287492].message }}
  • Profile picture of the author John Alves
    That's what the bot should do. However, it rarely indexes all the links at once. If you left them there long enough, it will come back to the page and eventually get them all. However, it would need some time to happen. I think Google does this to prevent spam. A lot of people would keep putting up posts and getting them indexed quick to try and get as many pages indexed as quickly as possible.
    Signature
    Quality SEO Articles on Any Topic in 24 Hours!
    {{ DiscussionBoard.errors[3287507].message }}
  • Profile picture of the author NeilC
    I don't know for sure but because links pages are not "liked" by Google the value of a link from one is not worth very much in most cases, although there are exceptions such as the quality directories.

    I believe that Google's algo is clever enough to distinguish between these and being as the links on the links page are seen as low value the bots probably don't waste time crawling them that often.

    Also remember that the bots will only spend very little time on a website unless it's a large authority site and don't crawl all content on every visit.

    I'd not worry about it too much and just get on with your website, if the links are worth much then they'll get found anyway. It's not uncommon for only about 10-20% of links to show depending on how you get them.

    Hope that help a bit,
    Neil
    Signature

    Need a fresh start or help to take your business to the next level? Click here to find out more...

    {{ DiscussionBoard.errors[3287567].message }}
    • Profile picture of the author dp40oz
      Originally Posted by NeilC View Post

      I don't know for sure but because links pages are not "liked" by Google the value of a link from one is not worth very much in most cases, although there are exceptions such as the quality directories.

      I believe that Google's algo is clever enough to distinguish between these and being as the links on the links page are seen as low value the bots probably don't waste time crawling them that often.

      Also remember that the bots will only spend very little time on a website unless it's a large authority site and don't crawl all content on every visit.

      I'd not worry about it too much and just get on with your website, if the links are worth much then they'll get found anyway. It's not uncommon for only about 10-20% of links to show depending on how you get them.

      Hope that help a bit,
      Neil
      Yea I guess im just trying to understand how the whole crawling thing works sometimes. I just feel the indexing of links seems so arbitrary. I'll spend time building 100's of quality links through good article posts and thought out blog comments on respected sites. Then a month later only a handful of those are indexed but a ton of forum profile signatures I did from forums nobody ever goes to are all indexed. Just confusing. I guess they like to keep it that way
      Signature
      {{ DiscussionBoard.errors[3287617].message }}
      • Profile picture of the author JohnMcCabe
        Originally Posted by dp40oz View Post

        Yea I guess im just trying to understand how the whole crawling thing works sometimes. I just feel the indexing of links seems so arbitrary. I'll spend time building 100's of quality links through good article posts and thought out blog comments on respected sites. Then a month later only a handful of those are indexed but a ton of forum profile signatures I did from forums nobody ever goes to are all indexed. Just confusing. I guess they like to keep it that way
        One thing to consider is that the googlebot doesn't crawl links, it crawls pages. And as I understand it, it's programmed to only spend a certain amount of time on any page. If it can't get everything in the time allotted, it takes what it can get and gets more on the next pass. It may follow links, or simply add them to a queue to follow at a later time.

        That's one reason why clean code and fast-loading pages are gaining in importance.

        Another thing to consider is, even if a page with your link (like a forum profile) is indexed, it may carry little to no actual weight.

        From time to time, you'll see people making posts asking why page x is outranking their page, even though they have 18 zillion backlinks and page x only has 3. Absent some other factor, it's because those three links carry more weight than the 18 zillion.

        Some search engine authorities have identified over 200 factors they believe go into the ranking algorithm, and sheer number of backlinks is only one of them. One which, if you believe recent news, will be waning in importance as the SEs continue the arms race against those who would manipulate the rankings using black hat methods.
        {{ DiscussionBoard.errors[3287705].message }}
  • Profile picture of the author Eliot Proud
    Hi

    If it was that easy then everyone would just build linkfarms and retire to warmer climes.

    I think there is too much emphasis put on creating backlinks.

    I have a site that only has five pages and some of the pages aren't finished and it has NO backlinks.

    It ranks number four on google for the keyword phrase that I chose and is only beaten by three big sites one being Wikipedia.

    Just my thoughts
    Eliot
    {{ DiscussionBoard.errors[3287576].message }}

Trending Topics