argh! where is this DAMN Robots.txt file!? -->SEO <--

Online Business Mentor · 12-18-2009, 10:47 AM

Hey warriors and friends....

It is so annoying......I have been trying to use a number of ways to seo one of my niche sites up the search engine ranks...

one of my SEO freelancers who helps has pointed out to me I have to remove the robot's txt file...which is as follows:

http://www.primovacationsclubreview.com/robots.txt (this is the domain am trying to seo)

But where is it!?....I have time and time again back and forward with my host (hostgator) and with including manual checked myself confirmed no file as such exists on my server....

I have then gone ahead into wordpress itself and looked around making sure search engines can crawl my bog/site...

but why is it , it still seems its being not indexed by google?...whats going on..?

I have even given it some social boomarking packets to give it a little boost

Elle Holder · 12-18-2009, 11:01 AM

I had some weirdness happen a few months ago with every new blog I created. And yes, it was just after I switched to HostGator.

I would add the site to Webmaster Tools and it would tell me that my robots.txt was set to disallow. Thing was, I had never created a robots.txt for any of these sites. I'd check the root file via FTP, and nope, no robots.txt there. I checked my privacy settings, and they were set properly too.

I corrected the problem by actually creating a robots.txt with user agent allow and uploaded that to my root. When Google found that, it canceled out the disallow that could not be found anywhere.

PS! Did you check your privacy settings in WP?

Online Business Mentor · 12-18-2009, 11:17 AM

thanks for your reply..much appreciated

yes privacy settings was something I checked on recently..and was secretly hoping it resolves it.

am not sure yet as it certainly hasn't seemed to.maybe takes some time...

LJNetworking · 12-18-2009, 11:24 AM

By clicking on the link that it is placed in your post, the file appears to be located in your public_html file. Ftp to your server and search for the file in that folder.

As far as your site not being indexed, there could be many reasons. To get your site indexed quickly, bookmarking your site is a great way to go because bookmarking sites are crawled constantly by Google. Sites like digg.com

Of course there are many other ways to get your site indexed, for example, posting to article directories and plug your site in the signature.

Get your site out there and give it some time and your site will be indexed.

Online Business Mentor · 12-18-2009, 11:31 AM

great tips..thats what i have done..some social bookmarks to boot it up...

amazingly cannot find such a file on the index page...so its really confusing..i'd have thot it turns into a 404 page when it doesnt work

KirkMcD · 12-18-2009, 01:25 PM

Quote:

Originally Posted by Online Business Mentor

but why is it , it still seems its being not indexed by google?...whats going on..?

You are indexed by google. When I type your url in, your homepage is listed, but nothing else.

Do you have backlinks to your other pages?

GeorgR. · 12-18-2009, 01:38 PM

that robots.txt is fine since you don't disallow anything.

If you're on wordpress, also check your privacy settings.

Currently it seems Google takes longer to actually pick up and index sites.

Online Business Mentor · 12-18-2009, 02:09 PM

Yes indexed but doesn't seem to want to bring the site anywhere unless I manually type it in...it has got backlinks to the index...not others..

Quote:

Originally Posted by KirkMcD

You are indexed by google. When I type your url in, your homepage is listed, but nothing else.

Do you have backlinks to your other pages?

palms · 12-18-2009, 03:06 PM

If you look at your file structure on a Wordpress site, you won’t find a physical robots.txt file. By default, WordPress internally generates a robots.txt each time a bot (or human) asks for it. So, what you are seeing is the result of an internal Wordpress script, not a file physically located within your file structure.

Online Business Mentor · 12-18-2009, 03:26 PM

thanks for the feedback..

much appreciated...

quick question..how does one edit internal scripts or locate it as you suggest?

Quote:

Originally Posted by palms

If you look at your file structure on a Wordpress site, you won’t find a physical robots.txt file. By default, WordPress internally generates a robots.txt each time a bot (or human) asks for it. So, what you are seeing is the result of an internal Wordpress script, not a file physically located within your file structure.

palms · 12-18-2009, 05:26 PM

I don't know of anyone who ever edited the internal WP script, since most people simply use a WP plugin. You can accomplish anything you want to with a few clicks.

I use the "Meta Robots WordPress" plugin from yoast.com.

12-18-2009, 10:47 AM	#1
Online Business Mentor Waseem Mirza War Room Member Join Date: Oct 2009 Location: Glasgow UK Posts: 268 Thanks: 24 Thanked 50 Times in 17 Posts Social Networking Contact Info	argh! where is this DAMN Robots.txt file!? -->SEO <-- Hey warriors and friends.... It is so annoying......I have been trying to use a number of ways to seo one of my niche sites up the search engine ranks... one of my SEO freelancers who helps has pointed out to me I have to remove the robot's txt file...which is as follows: http://www.primovacationsclubreview.com/robots.txt (this is the domain am trying to seo) But where is it!?....I have time and time again back and forward with my host (hostgator) and with including manual checked myself confirmed no file as such exists on my server.... I have then gone ahead into wordpress itself and looked around making sure search engines can crawl my bog/site... but why is it , it still seems its being not indexed by google?...whats going on..? I have even given it some social boomarking packets to give it a little boost
	Make Money Online With Waseem Mirza *Internet Marketing Course* *- $1997 Value! - But Yours For Free?* Waseem Mirza Official - Who The Heck Is This Guy Anyway? Waseem Mirza Blog - Meet Me On Video Here + Free Gifts

12-18-2009, 11:01 AM	#2
Elle Holder HyperActive Warrior Join Date: Dec 2009 Location: Canada Posts: 259 Thanks: 94 Thanked 46 Times in 38 Posts	Re: argh! where is this DAMN Robots.txt file!? -->SEO <-- I had some weirdness happen a few months ago with every new blog I created. And yes, it was just after I switched to HostGator. I would add the site to Webmaster Tools and it would tell me that my robots.txt was set to disallow. Thing was, I had never created a robots.txt for any of these sites. I'd check the root file via FTP, and nope, no robots.txt there. I checked my privacy settings, and they were set properly too. I corrected the problem by actually creating a robots.txt with user agent allow and uploaded that to my root. When Google found that, it canceled out the disallow that could not be found anywhere. PS! Did you check your privacy settings in WP?

12-18-2009, 11:17 AM	#3
Online Business Mentor Waseem Mirza War Room Member Join Date: Oct 2009 Location: Glasgow UK Posts: 268 Thanks: 24 Thanked 50 Times in 17 Posts Social Networking Contact Info	Re: argh! where is this DAMN Robots.txt file!? -->SEO <-- thanks for your reply..much appreciated yes privacy settings was something I checked on recently..and was secretly hoping it resolves it. am not sure yet as it certainly hasn't seemed to.maybe takes some time...
	Make Money Online With Waseem Mirza *Internet Marketing Course* *- $1997 Value! - But Yours For Free?* Waseem Mirza Official - Who The Heck Is This Guy Anyway? Waseem Mirza Blog - Meet Me On Video Here + Free Gifts

12-18-2009, 11:24 AM	#4
LJNetworking Larry Stroup - Warrior War Room Member Join Date: Aug 2008 Location: USA Posts: 30 Thanks: 1 Thanked 4 Times in 4 Posts	Re: argh! where is this DAMN Robots.txt file!? -->SEO <-- By clicking on the link that it is placed in your post, the file appears to be located in your public_html file. Ftp to your server and search for the file in that folder. As far as your site not being indexed, there could be many reasons. To get your site indexed quickly, bookmarking your site is a great way to go because bookmarking sites are crawled constantly by Google. Sites like digg.com Of course there are many other ways to get your site indexed, for example, posting to article directories and plug your site in the signature. Get your site out there and give it some time and your site will be indexed.

12-18-2009, 11:31 AM	#5
Online Business Mentor Waseem Mirza War Room Member Join Date: Oct 2009 Location: Glasgow UK Posts: 268 Thanks: 24 Thanked 50 Times in 17 Posts Social Networking Contact Info	Re: argh! where is this DAMN Robots.txt file!? -->SEO <-- great tips..thats what i have done..some social bookmarks to boot it up... amazingly cannot find such a file on the index page...so its really confusing..i'd have thot it turns into a 404 page when it doesnt work
	Make Money Online With Waseem Mirza *Internet Marketing Course* *- $1997 Value! - But Yours For Free?* Waseem Mirza Official - Who The Heck Is This Guy Anyway? Waseem Mirza Blog - Meet Me On Video Here + Free Gifts

12-18-2009, 01:38 PM	#7
GeorgR. Senior Warrior Member War Room Member Join Date: Jun 2007 Location: Earth Posts: 6,061 Blog Entries: 16 Thanks: 72 Thanked 915 Times in 602 Posts Social Networking	Re: argh! where is this DAMN Robots.txt file!? -->SEO <-- that robots.txt is fine since you don't disallow anything. If you're on wordpress, also check your privacy settings. Currently it seems Google takes longer to actually pick up and index sites.
	* Affiliate Site Quick --> The Fastest & Easiest Way to Make Affiliate Sites!<-- -> VISIT www.1UP-SEO.com * <- Internet Marketing, SEO Tips, Reviews & More!! *** ARTICLE SPINNING SERVICE - Custom Hand-Made QUALITY Spun Articles! - Thread Here * * BEAT the PENGUIN with High Quality Manually Spun Articles - Don't Settle for Less! *

12-18-2009, 03:06 PM	#9
palms Warrior Member Join Date: Nov 2009 Posts: 11 Thanks: 2 Thanked 3 Times in 3 Posts	Re: argh! where is this DAMN Robots.txt file!? -->SEO <-- If you look at your file structure on a Wordpress site, you won’t find a physical robots.txt file. By default, WordPress internally generates a robots.txt each time a bot (or human) asks for it. So, what you are seeing is the result of an internal Wordpress script, not a file physically located within your file structure.
	Last edited by palms; 12-18-2009 at 03:08 PM. Reason: sp

12-18-2009, 05:26 PM	#11
palms Warrior Member Join Date: Nov 2009 Posts: 11 Thanks: 2 Thanked 3 Times in 3 Posts	Re: argh! where is this DAMN Robots.txt file!? -->SEO <-- I don't know of anyone who ever edited the internal WP script, since most people simply use a WP plugin. You can accomplish anything you want to with a few clicks. I use the "Meta Robots WordPress" plugin from yoast.com.