12 {{ upvoteCount | shortNum }}
12 {{ upvoteCount | shortNum }}

argh! where is this DAMN Robots.txt file!? -->SEO <--

by Online Business Mentor
10 replies
  • SEO
    • |
Hey warriors and friends....

It is so annoying......I have been trying to use a number of ways to seo one of my niche sites up the search engine ranks...

one of my SEO freelancers who helps has pointed out to me I have to remove the robot's txt file...which is as follows:

http://www.primovacationsclubreview.com/robots.txt (this is the domain am trying to seo)

But where is it!?....I have time and time again back and forward with my host (hostgator) and with including manual checked myself confirmed no file as such exists on my server....

I have then gone ahead into wordpress itself and looked around making sure search engines can crawl my bog/site...

but why is it , it still seems its being not indexed by google?...whats going on..?

I have even given it some social boomarking packets to give it a little boost
#>seo #< #argh #damn #file #robotstxt
  • Profile picture of the author Elle Holder
    I had some weirdness happen a few months ago with every new blog I created. And yes, it was just after I switched to HostGator.

    I would add the site to Webmaster Tools and it would tell me that my robots.txt was set to disallow. Thing was, I had never created a robots.txt for any of these sites. I'd check the root file via FTP, and nope, no robots.txt there. I checked my privacy settings, and they were set properly too.

    I corrected the problem by actually creating a robots.txt with user agent allow and uploaded that to my root. When Google found that, it canceled out the disallow that could not be found anywhere.

    PS! Did you check your privacy settings in WP?
    Signature

    {{ DiscussionBoard.errors[1515369].message }}
    • Profile picture of the author Online Business Mentor
      thanks for your reply..much appreciated

      yes privacy settings was something I checked on recently..and was secretly hoping it resolves it.

      am not sure yet as it certainly hasn't seemed to.maybe takes some time...
      Signature
      {{ DiscussionBoard.errors[1515437].message }}
  • Profile picture of the author LJNetworking
    By clicking on the link that it is placed in your post, the file appears to be located in your public_html file. Ftp to your server and search for the file in that folder.

    As far as your site not being indexed, there could be many reasons. To get your site indexed quickly, bookmarking your site is a great way to go because bookmarking sites are crawled constantly by Google. Sites like digg.com

    Of course there are many other ways to get your site indexed, for example, posting to article directories and plug your site in the signature.

    Get your site out there and give it some time and your site will be indexed.
    {{ DiscussionBoard.errors[1515463].message }}
    • Profile picture of the author Online Business Mentor
      great tips..thats what i have done..some social bookmarks to boot it up...

      amazingly cannot find such a file on the index page...so its really confusing..i'd have thot it turns into a 404 page when it doesnt work
      Signature
      {{ DiscussionBoard.errors[1515489].message }}
  • Profile picture of the author KirkMcD
    Originally Posted by Online Business Mentor View Post

    but why is it , it still seems its being not indexed by google?...whats going on..?
    You are indexed by google. When I type your url in, your homepage is listed, but nothing else.

    Do you have backlinks to your other pages?
    {{ DiscussionBoard.errors[1515885].message }}
    • Profile picture of the author Online Business Mentor
      Yes indexed but doesn't seem to want to bring the site anywhere unless I manually type it in...it has got backlinks to the index...not others..



      Originally Posted by KirkMcD View Post

      You are indexed by google. When I type your url in, your homepage is listed, but nothing else.

      Do you have backlinks to your other pages?
      Signature
      {{ DiscussionBoard.errors[1516020].message }}
  • Profile picture of the author GeorgR.
    that robots.txt is fine since you don't disallow anything.

    If you're on wordpress, also check your privacy settings.

    Currently it seems Google takes longer to actually pick up and index sites.
    Signature
    *** Affiliate Site Quick --> The Fastest & Easiest Way to Make Affiliate Sites!<--
    -> VISIT www.1UP-SEO.com *** <- Internet Marketing, SEO Tips, Reviews & More!! ***
    *** HIGH QUALITY CONTENT CREATION +++ Manual Article Spinning (Thread Here) ***
    Content Creation, Blogging, Articles, Converting Sales Copy, Reviews, Ebooks, Rewrites
    {{ DiscussionBoard.errors[1515935].message }}
  • Profile picture of the author palms
    If you look at your file structure on a Wordpress site, you won't find a physical robots.txt file. By default, WordPress internally generates a robots.txt each time a bot (or human) asks for it. So, what you are seeing is the result of an internal Wordpress script, not a file physically located within your file structure.
    {{ DiscussionBoard.errors[1516163].message }}
    • Profile picture of the author Online Business Mentor
      thanks for the feedback..

      much appreciated...

      quick question..how does one edit internal scripts or locate it as you suggest?

      Originally Posted by palms View Post

      If you look at your file structure on a Wordpress site, you won't find a physical robots.txt file. By default, WordPress internally generates a robots.txt each time a bot (or human) asks for it. So, what you are seeing is the result of an internal Wordpress script, not a file physically located within your file structure.
      Signature
      {{ DiscussionBoard.errors[1516228].message }}
  • Profile picture of the author palms
    I don't know of anyone who ever edited the internal WP script, since most people simply use a WP plugin. You can accomplish anything you want to with a few clicks.

    I use the "Meta Robots WordPress" plugin from yoast.com.
    {{ DiscussionBoard.errors[1516558].message }}

Trending Topics