How deep the spider/crawler goes

by 7 comments
Hi all,

I have a website that is only partially indexed (Google, Yahoo..),
the issue is that important pages with good keywords didn't get to be indexed.

Q. When the crawler get to a page with many images (i know it won't index the images) will it go deeper on that page? - i have a page with images, but also the page contain a link to important page that i want
the SE to index.
Q. my site has only 11 pages indexed out of 110. is it normal? what can i do to have more indexed pages. Currently the crawler index just the main
pages and didn't go deep enough. (I've added my site to SE ~10 months ago)

last Q. my site was visited a month ago - how can i "invite" the SE again and not wait for it? is there a way i can control it?


#internet marketing #crawler #deep #images #index pages #spider #spider or crawler
  • Profile picture of the author Avizyah
    any idea guys...common help, what's your take on it???
  • Profile picture of the author Ron Killian
    Are you talking indexed or ranked? It's been 10 months? Very odd if it's been that long, seems there must be something wrong with those pages. Usually the first pass they grab the index, then come back later for the rest of the site. But shouldn't take that long.

    If you have important pages you want indexed, they should be linked off the index. How deep are they?

    One way to help it would be to get back links to those deeper pages.

    Best way to get the SE spiders back to your site, or more often, is to update the site. The more you update, the more they should visit. Good way to let them know you've updated is to use RSS and ping or site map's that notify SE's your site has changed.
    • Profile picture of the author Avizyah
      Hi Ron,

      well, 10 months was a bit of a stretch. Currently we have low PR,
      However Google only indexed 10 pages (major pages) out of 110 we have.
      I have in place a sitemap that I subbmitted to Google and it was verified ok.

      We are selling financial software and I would like google to index most of my reports pages since we have important keywords there.
      for example(Asset allocation)
      we have the same reports as pure images pages and also as HTML pages
      (see )

      The HTML reports are kind of "burried" , you can get to them via the images pages as a link if the visitor like to see the report as HTML version.
      Just the same the reports are included in the sitemap as links.
  • Profile picture of the author Ralf Skirr
    You can distribute higher internal page rank by carefully planning your internal linking structure. Have more links pointing to your important pages than too other pages.

  • Profile picture of the author Ron Killian
    I really don't why the pages are not getting indexed, don't see anything that would prevent it, unless I missed something. Though you do have some odd meta tags.

    Part of the problem could be, as you said, some of the pages are buried. Why? If you want them to rank well, you need to get them linked off the index and maybe closer to the index. It could be because you site has too much depth, with little power or back links, that might be a big part of it.

    I hope you don't take this the wrong way, but even if you do get indexed, the site is not SEO'ed well at all. You keywords should be in the url or page name, for one thing. There is very little text on the pages I looked at, the search engines want content, and your not giving them any. With all your graphics and code, you content is probably mute, or at the very least over shadowed by the code.

    Some of the pages use "-" to separate the words in the page name (good idea, though still need keywords in the page name) while other don't. You use upper and lower case, can't say that affects SEO, but I've always used lower case only. Many of the pages use the same keywords, like I noticed Asset Allocation on several pages. Your index page, should have your keywords first, not your site name, for optimal seo. Plus the index page title looks a little spammy, planning is used three times.

    To be completely honest, if you not getting traffic anyways, maybe a complete over haul would be best? If so, keep everything as close to the index as possible.

    Your still going to need alot more linkbacks. Your in a highly competitive niche, your going to need some power to get noticed in the SE's.

    This is kind of off-topic, but it might be good to re-vamp your index or main page, your trying to sell the features when you should be selling the benefits.

    Was I of any help?
    • Profile picture of the author Avizyah
      Yes, Very much so and I’ve started addressing some of the issues. The original designer of the website was not thinking at all in terms of web-marketing and SEO. So I’m in the process of making it more SEO friendly and already addressed some of the issues that I can put my hands on while trying not to sacrifice the style and design too much. My goal is to have good SEO site without sacrificing the look of it, the company likes the look and style so I have to take it into account.
      just the same, would you guys post few links of web-sites that you think have a good SEO implementation and are SEO oriented sites.

  • Profile picture of the author jensrsa
    I see you have about 80 pages index (Google search:

    - Make sure that your internal linking and navigation is in place
    - Use descriptive text links
    - Create and submit a google sitemap
    - Add a sitemap to your site
    - Improve your page titles to include keywords
    - Your pre-page to the frame, ExecPlan Express overview.htm, has meaningless text as far as your site content is concerned
    - The javascipt menu you use with the frames probably prevent spiders fro crawling those pages .
    - Move away from frames or at least use a text menu

    Just a few things


Next Topics on Trending Feed

  • 23 {{ upvoteCount | shortNum }} 5

    1. Lack Of Strategy Many of the tools available to Internet marketers are either cheap or free. As a result, many new marketers are tempted to use them all at once. Successful Internet marketers begin with a strategy that identifies the goals of a campaign, the target audience and other key planning elements. Only then do they choose their tactics. Remember: tactics without strategy is the noise before defeat.

  • 25 {{ upvoteCount | shortNum }} 5

    After five years and almost one thousand projects, give it or take, I believe I have come up with a handful of more or less reliable signs that you're dealing with a bad client. I haven't reinvented the wheel, that's for sure. It's just like in that saying. Maybe I don't know what I want, but I certainly know what I don't want. The same goes here. Maybe I don't know how to find a good client, but I certainly know how to avoid a bad one. Also, I'm far from an ideal list. Sharing experiences regarding bad clients can help a lot. After all, we play in the same team. In addition, I strongly believe in Good Karma principle. Meaning, there's plenty of room and clients for everybody. Deal with real people not with avatars

  • 9 {{ upvoteCount | shortNum }} 5

    When people think of the term copywriter they will often imagine someone sitting at home or at a caf, tapping away at a laptop and churning out good copy for their immediate employer whether it be a one-off project or a long-standing relationship between a publication (online or print) and the copywriter. Now that might often be correct, but there are different types of copywriters out there. Lets have a look at the main types.

  • 5 {{ upvoteCount | shortNum }} 5

    I've been a relatively successful Freelance writer on over the last several years. I haven'€™t worked totally consistently during that time, but I've done pretty well and have recently been growing my business pretty aggressively and turning up the heat! I recently got connected with a start-up company that frequently uses Freelance to get tasks done for cheap, and I had a great conversation with the founder of that company which serves as the inspiration for this article. Employers have to work hard to ensure that they'€™re getting the right person when they hire a Freelancer. Once a project is accepted and a milestone is created, it can be a major hassle to resolve any disputes, especially those surrounding the quality of the work. I don't have a lot of insight into this process but I have learned a thing or two about choosing the right employers. A lot of folks have it in their head that the customer is always right, but as a Freelancer, you have the ability to choose who is and isn'€™t going to be your customer. Here are some guidelines for finding great employers that fit with your business.