8 {{ upvoteCount | shortNum }}
8 {{ upvoteCount | shortNum }}

HELP! Search engines aren't seeing my pages!

by seosoldier
7 replies
Hi, I have had my site up for a few years and had 5 keywords on p1 of google. Suddenly though my site is not being seen by search engines. I mean that literally. When I try to make an xml site map like at xml-sitemaps.com, it only sees 2 pages, which are .pdf's, not my other 20 main pages.

I have a sitemap.html with all my links and I now have created my own xml sitemap (though not sure it's made correctly).

However, a guy at xml-sitemaps sent me this as an answer to my query. I don't know what to make of it except it looks very bad! :

"You can see how search engine robots view your site at http://www.xml-sitemaps.com/se-bot-simulator.html?op=se-bot-simulator&go=1&pageurl=http%3A%2F%2Fwhatsthebestwa terfilter.com%2F&se=googlebot&submit=Start"

If anyone can help me figure out what is wrong with my code, I'd really appreciate it. I plan to move my site to Wordpress soon but can't get to it right away and want the search engines to find and rank me in the mean time.

Thanks in advance to anyone who can give me any advice as to how to get my site back to health. I have no idea how this happened, I used to make site maps at the same site and it worked fine. (My site is in my signature below.)

(Could this have happened when I added the Wordpress page to my site? "Water Blog" in the left hand menu.)
#engines #pages #search
  • Profile picture of the author rhinocl
    Start with the obvious 3
    Check that a robots.txt file s not blocking them
    Go into the WordPress dashboard and look at privacy settings make sure they are not blocking search engines
    Lastly your pages aren't in a folder which requires a login are they?
    Signature
    cheap search engine optimization
    {{ DiscussionBoard.errors[6612444].message }}
    • Profile picture of the author seosoldier
      Originally Posted by rhinocl View Post

      Start with the obvious 3
      Check that a robots.txt file s not blocking them
      Go into the WordPress dashboard and look at privacy settings make sure they are not blocking search engines
      Lastly your pages aren't in a folder which requires a login are they?
      Hi, thanks for your reply.
      My site is not a Wordpress site.

      I checked the robots.txt file and 2 places I checked it at told me it's not the problem.

      The site's pages are all in my html folder in my CP as they always have been. None of the pages have been moved from when they were visible. Nothing has changed in that respect.

      I could be wrong but the only thing I can think of is that when I made some minor changes to the html /css some time ago, adding a widget, I may have deleted some necessary code by mistake? Of course I was careful and don't think that happened but that's the only thing I can think of.

      I am familiar with html but not with css. I didn't intend to mess with any of the CSS but maybe I did by mistake? Do you think this could be the cause?

      Do you have any ideas of where else besides here I can possibly get some help?

      The only thing I can think of is to rebuild the home page from scratch and hope that solves it...?
      Signature
      > My Promise To You: I will never promote any offer I do not truly believe to be 100% worth buying and using!
      https://bestwaterfilter.us
      {{ DiscussionBoard.errors[6615415].message }}
  • Profile picture of the author locke815
    Robot.txt it’s simply a file. But there is one interesting thing about it. It isn’t displayed to the actual visitors anywhere on the blog itself.
    Instead, it sits in the root directory of the blog and serves only one purpose. It is the file that search engines look at before they start crawling the contents of a blog. And the reason for looking at it is to find information on what they should and shouldn’t be crawling.
    So in essence, by using this file you can inform search engines what you want them to index and rank, and what you DON’T want them to index and rank.
    The truth is that not every page (or area) of a blog is worth ranking. As a webmaster or a person working with WordPress you have to be able to identify those areas and use robots.txt as a place where you can speak to search engines directly, and let them know what’s going on.
    {{ DiscussionBoard.errors[6615555].message }}
    • Profile picture of the author seosoldier
      Originally Posted by locke815 View Post

      Robot.txt it’s simply a file. But there is one interesting thing about it. It isn’t displayed to the actual visitors anywhere on the blog itself.
      Instead, it sits in the root directory of the blog and serves only one purpose. It is the file that search engines look at before they start crawling the contents of a blog. And the reason for looking at it is to find information on what they should and shouldn’t be crawling.
      So in essence, by using this file you can inform search engines what you want them to index and rank, and what you DON’T want them to index and rank.
      The truth is that not every page (or area) of a blog is worth ranking. As a webmaster or a person working with WordPress you have to be able to identify those areas and use robots.txt as a place where you can speak to search engines directly, and let them know what’s going on.
      Thank you, but I understand all that. My robots.txt file is not the problem.
      This is my robots.txt file:

      "
      # Allows all robots

      User-agent: *
      Sitemap: http://whatsthebestwaterfilter.com/sitemap.xml
      Disallow:
      "

      Before I did not have the Sitemap line in there, but it still did not give me a proper site map when trying to get one at Create your Google Sitemap Online - XML Sitemaps Generator and others which USED TO WORK JUST FINE.
      (I've tried a few different sitemap creators and all of them now show that I have just 2 pages, and both are pdf's! (?!)):

      "
      <?xml version="1.0" encoding="UTF-8"?>
      <urlset
      xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
      http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
      <!-- created with Free Online Sitemap Generator Create your Google Sitemap Online - XML Sitemaps Generator -->

      <url>
      <loc>http://whatsthebestwaterfilter.com/</loc>
      </url>
      <url>
      <loc>http://whatsthebestwaterfilter.com/Multi-PureWarranty.pdf</loc>
      <lastmod>2011-04-11T07:34:29+00:00</lastmod>
      </url>
      <url>
      <loc>http://whatsthebestwaterfilter.com/MultiPure880DataSheet.pdf</loc>
      <lastmod>2011-04-03T06:35:50+00:00</lastmod>
      </url>
      </urlset>
      "

      So for SOME unknown reason the bots are only seeing these 2 pages, NOT the 19 htm and html pages that are in my /site_map.htm

      I created my own .xml site map after realizing that the bots aren't seeing my pages, BUT this does not solve the problem as to why Google et al are not seeing my pages with their bot.
      Signature
      > My Promise To You: I will never promote any offer I do not truly believe to be 100% worth buying and using!
      https://bestwaterfilter.us
      {{ DiscussionBoard.errors[6615669].message }}
  • Profile picture of the author wayfarer
    Why do you think search engines don't "see" your site? Don't worry about silly sitemap tools. Clearly, Google "sees" your site. Just look at all the pages they have in their index: https://www.google.com/search?q=site...aterfilter.com
    Signature
    I build web things, server things. I help build the startup Veenome. | Remote Programming Jobs
    {{ DiscussionBoard.errors[6616941].message }}
  • Profile picture of the author wayfarer
    Anyway, besides my previous comment, the reason the sitemap tools don't see your site, is because your left navigation is created with JavaScript. Google figures this out eventually, since they read JavaScript eventually, but it is probably a good idea to update your menu with static HTML supplemented with JavaScript, not totally created with it.
    Signature
    I build web things, server things. I help build the startup Veenome. | Remote Programming Jobs
    {{ DiscussionBoard.errors[6616973].message }}
    • Profile picture of the author seosoldier
      Hi, and thanks for your response.
      Hmmm. I thought google wasn't seeing my site because I assumed that if the xml sitemap tools aren't seeing my site (even though they used to), then google probably isn't seeing it either. That, plus:

      Also according to google webmaster tools google has only 9 pages of my site's 21 pages indexed - this on a 21+ page site that has been up for about 3 years and I'm pretty sure they used to all be indexed.

      Also the Site Previews feature on google as of the other day was not seeing my site pages. I can't check right now but the other day it was not working.

      But it's good to see that at least 9 of my pages are indexed. Some of the pages you see on google are old pages that have been discontinued from my site. Guess I'm supposed to do a 301 or something on those, huh?

      Thanks for the tip on updating my menu. Now if only I can figure out how to do that! ;-D But as I said, the xml sitemaps sites used to work fine with the exact same menu and site, so something has changed... since now when I do a sitemap at 3 different sitemap creator sites, they only see 2 .pdf's. This concerns me because also I have lost all rankings and the only page ranked in the Top 5 pages of google is one of those pdf's. I had assumed I was hit by Penguin but now I wonder because the sitemaps only see the 2 pdf's and google is only ranking one of those pdf's.
      Signature
      > My Promise To You: I will never promote any offer I do not truly believe to be 100% worth buying and using!
      https://bestwaterfilter.us
      {{ DiscussionBoard.errors[6621387].message }}

Trending Topics