Search Engine is Not crawling my website?

13 replies
  • SEO
  • |
Hi,

I am facing a strange problem.

Search engine is not crawling my website completely.

Only static pages are crawled but business profile pages are not been crawled.

My website is https://www.vintelli.com/

Please suggest why its happening
#crawling #engine #search #website
  • Profile picture of the author nettiapina
    Originally Posted by nirmalya View Post

    Only static pages are crawled but business profile pages are not been crawled.
    If you're really talking about crawling, check your robots.txt and any indexing settings you may have in the actual pages. You're blogging some pages in the robots.txt, but none of those seems outright wrong (unless your /blog/ is an actual page and not just an example of some sort).

    Usually people are actually talking about pages being indexed. You seem to have several thousands of pages in the Google index. However, some of those are in the supplemental index because they're too similar to other content on your site.
    Signature
    Links in signature will not help your SEO. Not on this site, and not on any other forum.
    Who told me this? An ex Google web spam engineer.

    What's your excuse?
    {{ DiscussionBoard.errors[10082651].message }}
  • Profile picture of the author yasar
    check your site to using web master tool, then make a xml site map for your site, Dont used unwanted plugin
    {{ DiscussionBoard.errors[10082824].message }}
    • Profile picture of the author nettiapina
      Originally Posted by yasar View Post

      check your site to using web master tool, then make a xml site map for your site, Dont used unwanted plugin
      They have a xml sitemap, and you can clearly see from the robots.txt
      Signature
      Links in signature will not help your SEO. Not on this site, and not on any other forum.
      Who told me this? An ex Google web spam engineer.

      What's your excuse?
      {{ DiscussionBoard.errors[10082949].message }}
      • Profile picture of the author nirmalya
        Originally Posted by nettiapina View Post

        They have a xml sitemap, and you can clearly see from the robots.txt
        Yes, I have a XML sitemap but only statics pages are crawled there.

        No business pages have been crawled.
        {{ DiscussionBoard.errors[10083971].message }}
        • Profile picture of the author nettiapina
          Originally Posted by nirmalya View Post

          No business pages have been crawled.
          Are business pages are included in your nearly 9000 results? If yes, that's obviously not true. Google can't index before crawling.

          Are you actually saying that they don't show in your Google Webmaster Tools? Well, they're not included in the sitemaps you're providing to Google. Maybe there's a plugin to include everything from a network, but it's also possible to just code it.
          Signature
          Links in signature will not help your SEO. Not on this site, and not on any other forum.
          Who told me this? An ex Google web spam engineer.

          What's your excuse?
          {{ DiscussionBoard.errors[10085815].message }}
          • Profile picture of the author nirmalya
            Originally Posted by nettiapina View Post

            Are business pages are included in your nearly 9000 results? If yes, that's obviously not true. Google can't index before crawling.

            Are you actually saying that they don't show in your Google Webmaster Tools? Well, they're not included in the sitemaps you're providing to Google. Maybe there's a plugin to include everything from a network, but it's also possible to just code it.
            I have created xml sitemap using https://www.xml-sitemaps.com and only static pages are crawled there.
            That's the main problem I am facing.
            {{ DiscussionBoard.errors[10087857].message }}
            • Profile picture of the author nettiapina
              Originally Posted by nirmalya View Post

              That's the main problem I am facing.
              Well, if you know what the problem is then you just need to come up with a solution. I'd try one of the popular XML sitemap plugins, and see what it can do. Yoast SEO, for example, allows you to make custom sitemaps to include in the main one. You could just loop through network sites or whatever form content you're using.

              If this is your problem it doesn't have anything to do with crawling. Google is usually capable of doing it without your help. Your problem is that you're not seeing your articles in a certain report because of the limitations of your sitemap.xml. That's pretty much a non-issue.

              Originally Posted by SureshHiten

              One more thing that you can do that is start posting in quora, I ma sure within few you will get positive result.
              That has absolutely nothing to do with this thread.
              Signature
              Links in signature will not help your SEO. Not on this site, and not on any other forum.
              Who told me this? An ex Google web spam engineer.

              What's your excuse?
              {{ DiscussionBoard.errors[10089131].message }}
              • Profile picture of the author nirmalya
                [QUOTE=nettiapina;10089131]Well, if you know what the problem is then you just need to come up with a solution. I'd try one of the popular XML sitemap plugins, and see what it can do. Yoast SEO, for example, allows you to make custom sitemaps to include in the main one. You could just loop through network sites or whatever form content you're using.

                If this is your problem it doesn't have anything to do with crawling. Google is usually capable of doing it without your help. Your problem is that you're not seeing your articles in a certain report because of the limitations of your sitemap.xml. That's pretty much a non-issue


                Let me clear the details.

                I have created a new URL for the business pages and neither google is crawling the pages nor I can get the business pages included in xml sitemap.

                The website is : https://www.vintelli.com

                Please check the website and confirm me whats the main reason behind not crawling by sitemap generator as well as the search engine crawlers.
                {{ DiscussionBoard.errors[10091013].message }}
                • Profile picture of the author nettiapina
                  Originally Posted by nirmalya View Post

                  Let me clear the details.
                  What do you hope to achieve by repeating your original post? You brought no new information to the table.

                  You don't appear to have any problems with crawling. Your indexed page count? It's over 9000!

                  Or maybe a little less. Sorry, could not resist.

                  If you've got more articles than that, then say so. You've not even hinted at how many pieces you expect to be in the Google search index. But the indisputable fact is that Google crawls your site way beyond the static pages.

                  You apparently have a sitemap that doesn't cover what you hope it to cover. How about fixing this issue? Sitemaps allow sort of recursion, so you can have sitemaps within sitemaps. I'm not sure if Google WMT has some sort of size limitation, but you can at least try.

                  If you want to have some random external generator traverse your site, you may need to link to the category pages from the actual site structure. How do you expect the sitemap generator to actually find the profile pages? It's sort of a miracle that Google has apparently had no problems there.
                  Signature
                  Links in signature will not help your SEO. Not on this site, and not on any other forum.
                  Who told me this? An ex Google web spam engineer.

                  What's your excuse?
                  {{ DiscussionBoard.errors[10091286].message }}
  • Profile picture of the author cdebbie
    You can also check the speed of response from the server where your site is hosted, that sometimes can make Google to not to index your pages.
    {{ DiscussionBoard.errors[10083401].message }}
  • Profile picture of the author yukon
    Banned
    Originally Posted by nirmalya View Post

    Hi,

    I am facing a strange problem.

    Search engine is not crawling my website completely.

    Only static pages are crawled but business profile pages are not been crawled.

    My website is https://www.vintelli.com/

    Please suggest why its happening
    You have 9,000 pages indexed, I guess that's not enough.
    {{ DiscussionBoard.errors[10083405].message }}
  • Profile picture of the author WebLab
    Create an account at Google Webmaster Tools and add links of the pages which are not being crawled.
    {{ DiscussionBoard.errors[10084533].message }}
  • Profile picture of the author SureshHiten
    Make sure that your website is completely alright as per the SEO point of view i.e. robot.txt etc.
    One more thing that you can do that is start posting in quora, I ma sure within few you will get positive result.
    {{ DiscussionBoard.errors[10087940].message }}

Trending Topics