How deep the spider/crawler goes

by Avizyah 7 replies
Hi all,

I have a website that is only partially indexed (Google, Yahoo..),
the issue is that important pages with good keywords didn't get to be indexed.

Q. When the crawler get to a page with many images (i know it won't index the images) will it go deeper on that page? - i have a page with images, but also the page contain a link to important page that i want
the SE to index.
Q. my site has only 11 pages indexed out of 110. is it normal? what can i do to have more indexed pages. Currently the crawler index just the main
pages and didn't go deep enough. (I've added my site to SE ~10 months ago)

last Q. my site was visited a month ago - how can i "invite" the SE again and not wait for it? is there a way i can control it?


#main internet marketing discussion forum #crawler #deep #images #index pages #spider #spider or crawler
Avatar of Unregistered
  • Profile picture of the author Avizyah
    any idea guys...common help, what's your take on it???
    {{ DiscussionBoard.errors[116149].message }}
  • Profile picture of the author Ron Killian
    Are you talking indexed or ranked? It's been 10 months? Very odd if it's been that long, seems there must be something wrong with those pages. Usually the first pass they grab the index, then come back later for the rest of the site. But shouldn't take that long.

    If you have important pages you want indexed, they should be linked off the index. How deep are they?

    One way to help it would be to get back links to those deeper pages.

    Best way to get the SE spiders back to your site, or more often, is to update the site. The more you update, the more they should visit. Good way to let them know you've updated is to use RSS and ping or site map's that notify SE's your site has changed.
    PLR Affiliate Program Has Launched! Easily Promote Over 5,000 PLR and MRR Products.

    Largest Selection of PLR Articles on the Planet! PLR Ebooks, PLR Video, PLR Websites and more with Private Label Rights
    {{ DiscussionBoard.errors[116183].message }}
    • Profile picture of the author Avizyah
      Hi Ron,

      well, 10 months was a bit of a stretch. Currently we have low PR,
      However Google only indexed 10 pages (major pages) out of 110 we have.
      I have in place a sitemap that I subbmitted to Google and it was verified ok.

      We are selling financial software and I would like google to index most of my reports pages since we have important keywords there.
      for example(Asset allocation)
      we have the same reports as pure images pages and also as HTML pages
      (see )

      The HTML reports are kind of "burried" , you can get to them via the images pages as a link if the visitor like to see the report as HTML version.
      Just the same the reports are included in the sitemap as links.
      {{ DiscussionBoard.errors[116284].message }}
  • Profile picture of the author Ralf Skirr
    You can distribute higher internal page rank by carefully planning your internal linking structure. Have more links pointing to your important pages than too other pages.

    {{ DiscussionBoard.errors[116210].message }}
  • Profile picture of the author Ron Killian
    I really don't why the pages are not getting indexed, don't see anything that would prevent it, unless I missed something. Though you do have some odd meta tags.

    Part of the problem could be, as you said, some of the pages are buried. Why? If you want them to rank well, you need to get them linked off the index and maybe closer to the index. It could be because you site has too much depth, with little power or back links, that might be a big part of it.

    I hope you don't take this the wrong way, but even if you do get indexed, the site is not SEO'ed well at all. You keywords should be in the url or page name, for one thing. There is very little text on the pages I looked at, the search engines want content, and your not giving them any. With all your graphics and code, you content is probably mute, or at the very least over shadowed by the code.

    Some of the pages use "-" to separate the words in the page name (good idea, though still need keywords in the page name) while other don't. You use upper and lower case, can't say that affects SEO, but I've always used lower case only. Many of the pages use the same keywords, like I noticed Asset Allocation on several pages. Your index page, should have your keywords first, not your site name, for optimal seo. Plus the index page title looks a little spammy, planning is used three times.

    To be completely honest, if you not getting traffic anyways, maybe a complete over haul would be best? If so, keep everything as close to the index as possible.

    Your still going to need alot more linkbacks. Your in a highly competitive niche, your going to need some power to get noticed in the SE's.

    This is kind of off-topic, but it might be good to re-vamp your index or main page, your trying to sell the features when you should be selling the benefits.

    Was I of any help?
    PLR Affiliate Program Has Launched! Easily Promote Over 5,000 PLR and MRR Products.

    Largest Selection of PLR Articles on the Planet! PLR Ebooks, PLR Video, PLR Websites and more with Private Label Rights
    {{ DiscussionBoard.errors[116349].message }}
    • Profile picture of the author Avizyah
      Yes, Very much so and I’ve started addressing some of the issues. The original designer of the website was not thinking at all in terms of web-marketing and SEO. So I’m in the process of making it more SEO friendly and already addressed some of the issues that I can put my hands on while trying not to sacrifice the style and design too much. My goal is to have good SEO site without sacrificing the look of it, the company likes the look and style so I have to take it into account.
      just the same, would you guys post few links of web-sites that you think have a good SEO implementation and are SEO oriented sites.

      {{ DiscussionBoard.errors[116585].message }}
  • Profile picture of the author jensrsa
    I see you have about 80 pages index (Google search:

    - Make sure that your internal linking and navigation is in place
    - Use descriptive text links
    - Create and submit a google sitemap
    - Add a sitemap to your site
    - Improve your page titles to include keywords
    - Your pre-page to the frame, ExecPlan Express overview.htm, has meaningless text as far as your site content is concerned
    - The javascipt menu you use with the frames probably prevent spiders fro crawling those pages .
    - Move away from frames or at least use a text menu

    Just a few things

    {{ DiscussionBoard.errors[116958].message }}
Avatar of Unregistered

Trending Topics