Criteria by which google will index a page

36 replies
  • SEO
  • |
I read somewhere that google will not automatically index a page just because it is in a sitemap, that it also has to be linked to by another page with a certain level of authority, are you aware of any other criteria to increase indexability?
#criteria #google #index #page
  • Profile picture of the author Swapnil Tiwari
    Your on-page SEO must be on point, also other websites that link to your page is called a backlink.

    Having lots of high quality backlinks assures that your website gets a good index position in google.
    {{ DiscussionBoard.errors[11125266].message }}
    • Profile picture of the author zorbee
      [DELETED]
      {{ DiscussionBoard.errors[11125542].message }}
  • Profile picture of the author Risktaker89
    Include a few pages such as contact, about me / about us, privacy policy, income disclaimer to your website.

    Make sure your website is not infected with virus (mine was and it was the main reason it was not indexed)

    To help get your new post indexed, ping your blog post, share your link on twitter and google plus, submit your url to Google. This usually gets your post indexed on the same day.
    {{ DiscussionBoard.errors[11126312].message }}
  • Profile picture of the author nishagarg
    The best way to get your page indexed is to check your website's whois on several whois checker websites. They naturally crawl your website and gives Google a reason to index.

    Another way is to request indexing under the Google Console tool. It takes 1-5 hours for them to index a page. However, what I have discovered in this case is, since you forced to index a page, the ranking show no or inaccurate results.

    Even better, give your page a link from sites like Quora, LinkedIn, and WarriorForum (if it is niche related).

    Have tried all the 3 methods and all did work
    {{ DiscussionBoard.errors[11126391].message }}
    • Profile picture of the author nishagarg
      Also, as Risktaker89 mentioned, you should have 4 common pages which are about, contact, privacy policy, and disclaimer.
      {{ DiscussionBoard.errors[11126395].message }}
    • Profile picture of the author mattbarrie
      Would the reason that the whois server triggers google be that it creates a page with a backlink and then google thinks it's worth visiting?
      {{ DiscussionBoard.errors[11126400].message }}
      • Profile picture of the author nishagarg
        Yes, that's the case. I tested different ways for my different websites, and this can be the possible reason.
        {{ DiscussionBoard.errors[11126402].message }}
        • Profile picture of the author andre james
          I've heard this before but does it actually work? Also the backlink that is created does it remain or get deleted?
          {{ DiscussionBoard.errors[11128123].message }}
      • Profile picture of the author MikeFriedman
        Originally Posted by mattbarrie View Post

        Would the reason that the whois server triggers google be that it creates a page with a backlink and then google thinks it's worth visiting?
        Matt,

        It doesn't really do that. Whatever results nishagarg saw was just a coincidence.

        Plus, even if it did, it won't help you get a large site indexed or new pages of content.

        And the Search Console thing he mentioned is fine for a small site. For a very large one, it won't do as much good. You are limited how many times you can request Google to crawl pages on the site. Plus, it is very time consuming if you have say thousands of pages to enter in.
        Signature

        For SEO news, discussions, tactics, and more.
        {{ DiscussionBoard.errors[11126761].message }}
        • Profile picture of the author mattbarrie
          Mike, I've read somewhere that despite being a sitemap, Google will only index a page if it has another page pointing to it with enough authority. Thus I think when nishagarg used these whois sites, perhaps one of them made a page (e.g. domaintools) and linked back to the domain which kicked off the indexing process? Alternately could be a coincidence as you mention.
          {{ DiscussionBoard.errors[11126778].message }}
          • Profile picture of the author MikeFriedman
            Originally Posted by mattbarrie View Post

            Mike, I've read somewhere that despite being a sitemap, Google will only index a page if it has another page pointing to it with enough authority. Thus I think when nishagarg used these whois sites, perhaps one of them made a page (e.g. domaintools) and linked back to the domain which kicked off the indexing process? Alternately could be a coincidence as you mention.

            If that were true, brand new sites would almost never, ever get indexed.

            New Tumblr blogs would struggle to get indexed.

            I could go on and on with examples, but I think you get the idea.
            Signature

            For SEO news, discussions, tactics, and more.
            {{ DiscussionBoard.errors[11126838].message }}
            • Profile picture of the author mattbarrie
              New Tumblr blogs would be linked to from a directory or "new sites" page somewhere on Tumblr.
              {{ DiscussionBoard.errors[11127128].message }}
            • Profile picture of the author mattbarrie
              Also perhaps google checks the root / of all domains it knows about periodically?
              {{ DiscussionBoard.errors[11127129].message }}
        • Profile picture of the author AlphaAlpha
          Are you saying that we should all just disregard everything in nishararg's post?
          {{ DiscussionBoard.errors[11127027].message }}
          • Profile picture of the author MikeFriedman
            Originally Posted by AlphaAlpha View Post

            Are you saying that we should just disregard everything in nishararg's post?
            Pretty much.

            The Search Console thing works if you are just concerned with indexing a small static website.

            Building links from popular sites has a chance to bring the search engine spiders faster, but there is no guarantee on that. It depends on the crawling schedule, how deep they crawl, and when the last time was they crawled the site. And even then, just crawling a page doesn't mean Google is going to index it.

            If you are having trouble getting pages indexed, there is most likely a problem with the site.
            Signature

            For SEO news, discussions, tactics, and more.
            {{ DiscussionBoard.errors[11127045].message }}
            • Profile picture of the author nishagarg
              @MikeFriedman, I agree that this can be a coincidence. This is true that my websites are WordPress based and have 20-100 pages. Perhaps a small website.

              Another best thing you can do mattbarrie is add small unique posts at sites like SteemIt, OnMogul, Medium and Tumblr.
              {{ DiscussionBoard.errors[11127175].message }}
  • Profile picture of the author luxazan
    Try to Have Unique TITLE and DESCRIPTION and of course worthy content. that is all google needs to index a page.
    {{ DiscussionBoard.errors[11126715].message }}
    • Profile picture of the author mattbarrie
      This could be a reason why some pages are not being indexed.
      {{ DiscussionBoard.errors[11126779].message }}
  • Profile picture of the author topcoder
    Even if google indexes your page, it doesn't mean you'll get traffic from the search giant. That's where SERP comes into play.
    {{ DiscussionBoard.errors[11127063].message }}
    • Profile picture of the author mattbarrie
      Yes of course but I am just talking about basic indexability at this point
      {{ DiscussionBoard.errors[11127133].message }}
  • Profile picture of the author yukon
    Banned
    Google will index a page without any links pointing at the page/URL.

    An xml sitemap alone will get a page indexed.

    Strictly talking indexing here, not ranking.
    {{ DiscussionBoard.errors[11127141].message }}
    • Profile picture of the author mattbarrie
      Google will not index everything in the sitemap, I am trying to figure out the conditions by which Google will decide to index something from that sitemap or not
      {{ DiscussionBoard.errors[11127143].message }}
      • Profile picture of the author yukon
        Banned
        Originally Posted by mattbarrie View Post

        Google will not index everything in the sitemap, I am trying to figure out the conditions by which Google will decide to index something from that sitemap or not

        Google will index URLs in an xml sitemap.

        Impossible to know why your sitemap is failing without seeing the file.

        You also need to include the xml sitemap URL/s in the robots.txt file.
        {{ DiscussionBoard.errors[11127837].message }}
        • Profile picture of the author Mike Anthony
          Originally Posted by yukon View Post


          Impossible to know why your sitemap is failing without seeing the file.
          I might have got it wrong from this or some other related thread but I got that Matt was talking about a fairly large site.
          Signature

          {{ DiscussionBoard.errors[11127840].message }}
          • Profile picture of the author yukon
            Banned
            Originally Posted by Mike Anthony View Post

            I might have got it wrong from this or some other related thread but I got that Matt was talking about a fairly large site.


            I read the other thread, 40 million pages.

            Big project but doable. That's roughly 800 child sitemaps considering Google will process files up to 50MB and 50,000 URLs.

            Actually this could be OPs problem If he's got files larger than the Google max. Maybe Google hits the max limit on OPs file/s and bails before processing the entire file/s.

            Again, hard to help without seeing OPs xml files.
            {{ DiscussionBoard.errors[11127851].message }}
            • Profile picture of the author Mike Anthony
              Originally Posted by yukon View Post

              I read the other thread, 40 million pages.

              Big project but doable. That's roughly 800 child sitemaps considering Google will process files up to 50MB and 50,000 URLs.

              Actually this could be OPs problem If he's got files larger than the Google max. Maybe Google hits the max limit on OPs file/s and bails before processing the entire file/s.
              Possible but I think this gets down tot he area of disagreement between the two of you. I don't see anywhere where Google has promised to keep pages indexed merely because they are submitted in a sitemap. In fact not even using my own anecdotal evidence its been a widely reported phenomenon across the web that google drops lots of pages from larger sites.

              With so many pages some may not be considered to be worthy of indexing (like duplicates etc).

              I'd definitely agree its impossible to figure out without seeing it.
              Signature

              {{ DiscussionBoard.errors[11127872].message }}
              • Profile picture of the author yukon
                Banned
                Originally Posted by Mike Anthony View Post

                Possible but I think this gets down tot he area of disagreement between the two of you. I don't see anywhere where Google has promised to keep pages indexed merely because they are submitted in a sitemap. In fact not even using my own anecdotal evidence its been a widely reported phenomenon across the web that google drops lots of pages from larger sites.

                With so many pages some may not be considered to be worthy of indexing (like duplicates etc).

                I'd definitely agree its impossible to figure out without seeing it.


                If it's duplicate pages for OP it's another story which could be getting tripped up in an entirely different algo. (indexed by one algo... deindexed by another algo.).

                Sure nothing is guaranteed with Google but there's best practices, well, reverse engineering other sites to get what you want.

                As far as finding sitemaps, it needs to be easily accessible...
                • Build valid sitemap/s
                • Add sitemap URL to robots.txt
                • Submit robots.txt to Google
                {{ DiscussionBoard.errors[11127881].message }}
            • Profile picture of the author mattbarrie
              For those that want to look at the sitemap, they are listed in here: https://www.freelancer.com/robots.txt
              {{ DiscussionBoard.errors[11129786].message }}
        • Profile picture of the author mattbarrie
          Google will not index everything in a sitemap, they say so explicitly and they also show the % in webmaster tools.
          {{ DiscussionBoard.errors[11129785].message }}
      • Profile picture of the author Mike Anthony
        Originally Posted by mattbarrie View Post

        Google will not index everything in the sitemap, I am trying to figure out the conditions by which Google will decide to index something from that sitemap or not
        I think there is some communicating at cross purposes here. When people usually say "getting a page indexed" they are usually talking about their first page/pages of a site. Yukon is right. I can't think of any time I have started any blog that it has not been indexed and often with no authoritative backlink (or even a sitemap). Personally I have never understood the great angst with indexing....indexed on page 20 for a term someone searches is just as lost in space as not being indexed so I focus on ranking.

        However what you are talking about is getting hundreds/thousands of pages indexed. Thats a different dynamic. A good while ago Google admitted it reserves a certain amount of crawl time for each site/entry point. The bot goes through indexes and then depending on how authoritative the site is (by incoming links - and whatever secret sauce google has) it decides to leave at a certain point. Makes perfect sense as crawl time is a resource Google is eating up and someone could create a 100 million long page gibberish site and appreciably slow Google's operations.

        So if you have a smallish site you are likely to get all the pages indexed and fairly easily. The bot will come in through some link (really any generic link of low authority or not) do a minimum depth crawl and be on its way getting everything. If your site is huge and more time and resources are going to be used than the minimum then there has to be factors to that site (or entry point) that makes it hang around longer and go deep.

        Authoritative link would do it. New content is another factor. Google crawls sites that change often more frequently than those that don't change at all.

        So I think both your points are valid - you are just talking about slightly different things.
        Signature

        {{ DiscussionBoard.errors[11127838].message }}
  • Profile picture of the author Anand Kumar Kush
    create backlinks for your website
    {{ DiscussionBoard.errors[11127319].message }}
  • Profile picture of the author mrjackpowers
    I've had plenty of pages indexed which were never on a site map nor connected to other pages. Although, I do recommend both. The quickest way to get your pages indexed is to submit your site to Google AND start getting clicks from offsite links as soon as possible.
    {{ DiscussionBoard.errors[11127336].message }}
  • Profile picture of the author shipwrecked
    The sitemap helps Google better understand the structure of the website, but it might not help indexing it... In the past they said that placing AdSense ads on a site helps get indexed faster!
    {{ DiscussionBoard.errors[11127455].message }}
  • I personally have had success buy submitting my blog entries and sites to pingomatic and adding quality back links. Personally I found the safest and easiest backlinks to be from multiple niche related social profiles: pintrest, facebook, twitter, tumblr. Also reddit is good too....
    {{ DiscussionBoard.errors[11127620].message }}
  • Profile picture of the author TrafficQueen
    Banned
    I would be placing as many backlinks as i could.
    {{ DiscussionBoard.errors[11127960].message }}
  • Profile picture of the author netgeek
    Google doesn't index a page automatically.

    But, your posted page does get indexed within minutes because your website url is indexed on other website pages like whois database websites.

    When Google bots crawl those whois database pages, they come to your website to crawl, gather the data and index.

    Increase Indexability?
    Any new page posted, you can submit google index using your webmaster tools, and indexing happens within seconds, may take minutes in some cases though.

    Let me know if you still need any info.
    {{ DiscussionBoard.errors[11127986].message }}
  • Profile picture of the author Alireza Safiri
    Banned
    [DELETED]
    {{ DiscussionBoard.errors[11128530].message }}
  • Profile picture of the author mattbarrie
    For those that want to look at the sitemap, they are listed in here: https://www.freelancer.com/robots.txt
    {{ DiscussionBoard.errors[11129787].message }}

Trending Topics