13 {{ upvoteCount | shortNum }}
13 {{ upvoteCount | shortNum }}

Indexing with Google

by aaronjunited
11 replies
  • SEO
    • |
I really really need help with this.

I have been searching for days now for a solution to my problem.
Problem: Google has indexed my entire WordPress directory, all of the plugins and wpcontent folder.

I have a robots.txt file set up to block the indexing of folders.

It's not working.
Also I removed the entire site with Remove URL tool on webmaster tools.

I then inserted the robots.txt file and re included the site, but Google has still indexed my entire directory. 200 odd results in fact.


I desperately need help.
Can someone add me on Skype or whatever and help me out.

Thanks.
#google #indexing
  • Profile picture of the author Adie
    Install Google XML Sitemap
    Go to Google webmaster tools and submit your sitemap

    problem solved....
    No need to add any robot code or something. Google will only index whatever you have in your sitemap...
    Signature



    Moderator's Note: You're only allowed to put your own products or sites in your signature.

    Signature edited.
    {{ DiscussionBoard.errors[6762163].message }}
  • Profile picture of the author Chri5123
    ^ This!

    I think site maps will help sort all of this out.

    However even without site maps not sure why it has indexed everything!

    How long have you had the site set up?

    Chris
    {{ DiscussionBoard.errors[6762172].message }}
  • Profile picture of the author johan_malmo
    Great solution if it's enough! Easy to do too...
    Signature
    {{ DiscussionBoard.errors[6762173].message }}
  • Profile picture of the author aaronjunited
    Been there done that, installed Google XML sitemap plugin and it created a second sitemap, as I use SEO Yoast, it provides a sitemap.

    I have submitted my sitemap with Google and there is 3 different sitemaps that Yoast created, page, post and category.

    There is an error for each in the sitemap section on webmaster tools saying URL blocked by robots.txt

    Do I need to give Google time to de-index my directory?
    {{ DiscussionBoard.errors[6762181].message }}
    • Profile picture of the author Adie
      Originally Posted by aaronjunited View Post

      Been there done that, installed Google XML sitemap plugin and it created a second sitemap, as I use SEO Yoast, it provides a sitemap.

      I have submitted my sitemap with Google and there is 3 different sitemaps that Yoast created, page, post and category.

      There is an error for each in the sitemap section on webmaster tools saying URL blocked by robots.txt

      Do I need to give Google time to de-index my directory?
      Use all-in-one seo for your seo.
      use google xml sitemap for sitemap
      remove any existing sitemap in google webmaster tools

      I had this problem before (multiple times while fixing customers' sites) but they were sorted out with the above steps without any problem.
      Signature



      Moderator's Note: You're only allowed to put your own products or sites in your signature.

      Signature edited.
      {{ DiscussionBoard.errors[6762189].message }}
  • Profile picture of the author aaronjunited
    I have been building the site, it's not finished and Google indexed the directory while I was building the site.

    I had privacy option turned on in WordPress while doing this, but I decided to check site: bla bla.com one day and it threw up my entire directory.

    I've been building over the last week or two.
    {{ DiscussionBoard.errors[6762185].message }}
  • Profile picture of the author aaronjunited
    So back to the old debate on SEO Yoast vs All in one SEO lol, sorry.

    Ok so get rid of the plugin then? Remove the sitemap from Google, then resubmit it?

    I take it Yoast is causing this problem?
    {{ DiscussionBoard.errors[6762200].message }}
    • Profile picture of the author Adie
      Originally Posted by aaronjunited View Post

      So back to the old debate on SEO Yoast vs All in one SEO lol, sorry.

      Ok so get rid of the plugin then? Remove the sitemap from Google, then resubmit it?

      I take it Yoast is causing this problem?
      Try it first and see the result. I'm not really particular on what SEO plugin to use as long as google can read them.... I have no problem at all. I only use all in one seo ever since... and not planning to use other similar plugins..
      Signature



      Moderator's Note: You're only allowed to put your own products or sites in your signature.

      Signature edited.
      {{ DiscussionBoard.errors[6762211].message }}
  • Profile picture of the author vgvetter
    Have been fighting confusing results shown by Webmaster Tools for the last four months.

    Monday everything indexed....Friday nothing indexed...but half my traffic, about 100/month still comes thru search.

    Have concluded that the data at WMT is not always currant... it may reflect site changes instantly and at other times it may take weeks to show difference. If this is be caused by a plugin haven't figured it out. So, don't waste too much time worrying about it. Concentrate on other traffic sources.
    {{ DiscussionBoard.errors[6762447].message }}
  • Profile picture of the author MatthewWoodward
    In your .htaccess file add-

    Code:
    Options -Indexes
    This will stop the folders returning an index when you hit them via URL directly and in due time Google will remove them all from the index.
    Signature
    {{ DiscussionBoard.errors[6763052].message }}
  • Profile picture of the author Adie
    I checked his site and there is a problem with the server. The server is not accepting any sitemap and any plugin does not generating any sitemap as well. I even uploaded blank sitemaps manually but google xml sitemap still not reading it. It looks like his server is rejecting permissions. I suggested him to reinstall everything or contact hostgator support...
    Signature



    Moderator's Note: You're only allowed to put your own products or sites in your signature.

    Signature edited.
    {{ DiscussionBoard.errors[6763084].message }}

Trending Topics