Indexing with Google

by 11 replies
13
I really really need help with this.

I have been searching for days now for a solution to my problem.
Problem: Google has indexed my entire WordPress directory, all of the plugins and wpcontent folder.

I have a robots.txt file set up to block the indexing of folders.

It's not working.
Also I removed the entire site with Remove URL tool on webmaster tools.

I then inserted the robots.txt file and re included the site, but Google has still indexed my entire directory. 200 odd results in fact.


I desperately need help.
Can someone add me on Skype or whatever and help me out.

Thanks.
#search engine optimization #google #indexing
  • Install Google XML Sitemap
    Go to Google webmaster tools and submit your sitemap

    problem solved....
    No need to add any robot code or something. Google will only index whatever you have in your sitemap...
  • ^ This!

    I think site maps will help sort all of this out.

    However even without site maps not sure why it has indexed everything!

    How long have you had the site set up?

    Chris
  • Great solution if it's enough! Easy to do too...
  • Been there done that, installed Google XML sitemap plugin and it created a second sitemap, as I use SEO Yoast, it provides a sitemap.

    I have submitted my sitemap with Google and there is 3 different sitemaps that Yoast created, page, post and category.

    There is an error for each in the sitemap section on webmaster tools saying URL blocked by robots.txt

    Do I need to give Google time to de-index my directory?
    • [1] reply
    • Use all-in-one seo for your seo.
      use google xml sitemap for sitemap
      remove any existing sitemap in google webmaster tools

      I had this problem before (multiple times while fixing customers' sites) but they were sorted out with the above steps without any problem.
  • I have been building the site, it's not finished and Google indexed the directory while I was building the site.

    I had privacy option turned on in WordPress while doing this, but I decided to check site: bla bla.com one day and it threw up my entire directory.

    I've been building over the last week or two.
  • So back to the old debate on SEO Yoast vs All in one SEO lol, sorry.

    Ok so get rid of the plugin then? Remove the sitemap from Google, then resubmit it?

    I take it Yoast is causing this problem?
    • [1] reply
    • Try it first and see the result. I'm not really particular on what SEO plugin to use as long as google can read them.... I have no problem at all. I only use all in one seo ever since... and not planning to use other similar plugins..
  • Have been fighting confusing results shown by Webmaster Tools for the last four months.

    Monday everything indexed....Friday nothing indexed...but half my traffic, about 100/month still comes thru search.

    Have concluded that the data at WMT is not always currant... it may reflect site changes instantly and at other times it may take weeks to show difference. If this is be caused by a plugin haven't figured it out. So, don't waste too much time worrying about it. Concentrate on other traffic sources.
    • [ 1 ] Thanks
  • In your .htaccess file add-

    Code:
    Options -Indexes
    This will stop the folders returning an index when you hit them via URL directly and in due time Google will remove them all from the index.
  • I checked his site and there is a problem with the server. The server is not accepting any sitemap and any plugin does not generating any sitemap as well. I even uploaded blank sitemaps manually but google xml sitemap still not reading it. It looks like his server is rejecting permissions. I suggested him to reinstall everything or contact hostgator support...
    • [ 1 ] Thanks
Join the discussion

Next Topics on Trending Feed