5 {{ upvoteCount | shortNum }}
5 {{ upvoteCount | shortNum }}

Indexing Problem Advice Needed

by Maxtemple
4 replies
  • SEO
    • |
I am new here so excuse my errors, I hope this is the correct section for this. I thought I would ask for help.

I have a new website (on the WordPress blog platform) that I have indexed, the problem is that I tried to find out how many pages have been indexed and what I got was an the home page and an entry like this: User-agent: * Disallow: Sitemap:* from a robots .txt which I did not put on the site. I searched for this file but it's not there. I added my own robots.txt file allowing search engines to crawl the site and manually submitted the sitemap in my Google webmaster account.

The problem is that no matter what I do the number of pages indexed doesn't seem to change. In the webmaster account it says that other pages have been indexed but I can't tell which pages as they don't show up when you search using operators like "site:******".
I guess I just want to know if anyone has had this happen to them and is there anything that can be done about it?
#advice #google #indexed pages #indexing #needed #problem
  • Profile picture of the author dmtaylor247
    Firstly you searched for a robots.txt and wasn't there but you put it there?
    You indexed the site but nothing is appearing, you indexed it?

    Am I missing something?

    You can sometimes have conflicts with plugins that block crawlers or depending on the platform there may be additional settings.

    The robots.txt looks ok;

    User-agent: *
    Disallow:

    Sitemap:*

    But you could do with an additional Allow: / and a sitemap like this;

    User-agent: *
    Disallow:
    Allow: /

    Sitemap: http://www.example.com/sitemap.xml

    Where is the sitemap in this file that you submitted to GWT? Have you got an .xml or .gz sitemap?

    Sorry I'm lost..

    Re-check all of your wordpress settings to make sure it's indexing the site properly. It could be a setting instructing wordpress to alter the robots.txt file and not index the site.

    Then go to GWT, resubmit your sitemap, fetch your index page to test the crawlers picking up your robots.txt file correctly and bookmark your site a few times.
    {{ DiscussionBoard.errors[6313967].message }}
    • Profile picture of the author Maxtemple
      Originally Posted by dmtaylor247 View Post

      Firstly you searched for a robots.txt and wasn't there but you put it there?
      You indexed the site but nothing is appearing, you indexed it?

      Am I missing something?
      The first robots.txt file was not put there by me. I saw it first in the results when I did a search using the "site:" operator in Google. I went to the site to find it and it was not there. The only 2 pages that show in the results are the home page and this robot.txt page.
      In the GWT they show many other pages indexed, but these indexed pages do not show up in the search results.
      {{ DiscussionBoard.errors[6314397].message }}
      • Profile picture of the author dmtaylor247
        So you have loaded this file up in you browser and it's not there? it must be? Otherwise Google would have probably indexed your site.

        yourdomain.com/robots.txt

        Otherwise if you are looking on your host the files may be hidden, you need to enable all files before you enter the file manger and make sure you are looking in the correct folder.

        Have you checked the Wordpess privacy settings? There will be some sort of option like this; Allow search engines to index this site

        I think this is your problem.

        Settings Privacy Screen « WordPress Codex

        It alters your robots.txt file and adds extra noindex meta data the the <head> tag. You need to check through everything, otherwise go to your host and update the files yourself.

        If you right click your site, view the source code and CNTR + F and type in noindex and you find it then it's definately wordpress you need to change not your robots.txt file.
        {{ DiscussionBoard.errors[6314907].message }}
  • Profile picture of the author masterjani
    check your meta tags for any no index attributes, also delete any exisiting robots.txt and create a new one.
    Signature
    Increase your Google rank - (VARIETY SEO) High Quality Backlinks From Different Sources
    {{ DiscussionBoard.errors[6315202].message }}

Trending Topics