Indexing Problem Advice Needed

by 4 replies
5
I am new here so excuse my errors, I hope this is the correct section for this. I thought I would ask for help.

I have a new website (on the WordPress blog platform) that I have indexed, the problem is that I tried to find out how many pages have been indexed and what I got was an the home page and an entry like this: User-agent: * Disallow: Sitemap:* from a robots .txt which I did not put on the site. I searched for this file but it's not there. I added my own robots.txt file allowing search engines to crawl the site and manually submitted the sitemap in my Google webmaster account.

The problem is that no matter what I do the number of pages indexed doesn't seem to change. In the webmaster account it says that other pages have been indexed but I can't tell which pages as they don't show up when you search using operators like "site:******".
I guess I just want to know if anyone has had this happen to them and is there anything that can be done about it?
#search engine optimization #advice #google #indexed pages #indexing #needed #problem
  • Firstly you searched for a robots.txt and wasn't there but you put it there?
    You indexed the site but nothing is appearing, you indexed it?

    Am I missing something?

    You can sometimes have conflicts with plugins that block crawlers or depending on the platform there may be additional settings.

    The robots.txt looks ok;

    User-agent: *
    Disallow:

    Sitemap:*

    But you could do with an additional Allow: / and a sitemap like this;

    User-agent: *
    Disallow:
    Allow: /

    Sitemap: http://www.example.com/sitemap.xml

    Where is the sitemap in this file that you submitted to GWT? Have you got an .xml or .gz sitemap?

    Sorry I'm lost..

    Re-check all of your wordpress settings to make sure it's indexing the site properly. It could be a setting instructing wordpress to alter the robots.txt file and not index the site.

    Then go to GWT, resubmit your sitemap, fetch your index page to test the crawlers picking up your robots.txt file correctly and bookmark your site a few times.
    • [1] reply
    • The first robots.txt file was not put there by me. I saw it first in the results when I did a search using the "site:" operator in Google. I went to the site to find it and it was not there. The only 2 pages that show in the results are the home page and this robot.txt page.
      In the GWT they show many other pages indexed, but these indexed pages do not show up in the search results.
      • [1] reply
  • check your meta tags for any no index attributes, also delete any exisiting robots.txt and create a new one.
Join the discussion

Next Topics on Trending Feed

  • 5

    Indexing Problem Advice Needed

    I am new here so excuse my errors, I hope this is the correct section for this. I thought I would ask for help. I have a new website (on the WordPress blog platform) that I have indexed, the problem is that I tried to find out how many pages have been indexed and what I got was an the home page and an entry like this: User-agent: * Disallow: Sitemap:* from a robots .txt which I did not put on the site. I searched for this file but it's not there. I added my own robots.txt file allowing search engines to crawl the site and manually submitted the sitemap in my Google webmaster account.