11 {{ upvoteCount | shortNum }}
11 {{ upvoteCount | shortNum }}

Stopping Google from indexing page

by Oscarfishlover
10 replies
  • SEO
    • |
Sorry, I wasn't sure whether I should put this in the actual web design part, or the SEO.

Anyway, I'm trying to optimise a store that I run for my mates, basically we sell casual clothing. I have had a look at all the pages by typing in site:http://www.mysite.com and I have identified a few pages that although are important to the website, really shouldn't be appearing on the search engines. Therefore, I want to try and block as many of these pages as possible to see if I can improve the traffic to the website. I used Joomla! 2.5 for the website. However, should I be using my robot.txt file to block these individual pages?

Thanks in advance
#google #indexing #page #stopping
  • Profile picture of the author entrepenerd
    Robots.txt is going to be the quickest way to stop the bots from indexing the content (or at least including the indexed content in the SERPs).

    I've not used Joomla, but there may be an option to set a meta tag on the page in the Joomla system for noindex. If you can do that on a page by page basis, that would be good also.

    You'd want it to set a meta like:

    <meta name="robots" content="noindex">

    That in combination with the robots.txt updates should have you covered.
    Signature
    FREE Video Reveals...
    7 Secrets to Internet Marketing Failure
    ...or, how you can short circuit my mistakes and become an Internet Market Success!!
    {{ DiscussionBoard.errors[8923827].message }}
  • Profile picture of the author Oscarfishlover
    Thanks for the reply, I knew somebody on here would have an answer

    Okay, I'm just looking in the article section on the site. There is a section where you enter in your meta description, keywords etc. Underneath is a drop-down menu, is entitled robots. There doesn't appear to be anywhere where I can manually answer in information, however these are the options: index, follow | no index, follow | index, no follow | no index, no follow

    I would imagine that I would choose "no index, follow" In other words, it will still work, but it won't be indexed by Google. If I'm wrong, please tell me

    Some of the pages are not articles, so I'll probably have to use a robots.txt.

    Thanks
    {{ DiscussionBoard.errors[8923977].message }}
  • Profile picture of the author entrepenerd
    Yes, your noindex follow option should work for what you need.
    Signature
    FREE Video Reveals...
    7 Secrets to Internet Marketing Failure
    ...or, how you can short circuit my mistakes and become an Internet Market Success!!
    {{ DiscussionBoard.errors[8924064].message }}
    • Profile picture of the author Oscarfishlover
      the
      Originally Posted by entrepenerd View Post

      Yes, your noindex follow option should work for what you need.
      could you also tell me how to block individual pages, what the exact code I need to put into the robot.txt file? For instance, one of the pages that I don't really want coming up is the register page, if you search the site then various pages relating to your registration show, they don't need to show and I would rather stop them.

      Thanks
      {{ DiscussionBoard.errors[8924112].message }}
  • Profile picture of the author pikupiku
    Banned
    [DELETED]
    {{ DiscussionBoard.errors[8924145].message }}
    • Profile picture of the author paulgl
      There is no way to ever 100% guarantee that google will not
      index something unless you take it offline, or behind a password.

      There is no reason to block google from indexing a page from
      a normal, real, good, decent, money-making site. Especially
      in the context of what you are saying.

      Google is smart enough to know what to show in serps for
      people who are searching.

      Google has been at this a long, long time, and has the best
      engineers around.

      Nobody in their right search-mind is ever going to see those pages
      in serps, much less click on the dang things, even if they did!

      Paul
      Signature

      If you were disappointed in your results today, lower your standards tomorrow.

      {{ DiscussionBoard.errors[8924169].message }}
      • Profile picture of the author Oscarfishlover
        Originally Posted by paulgl View Post

        There is no way to ever 100% guarantee that google will not
        index something unless you take it offline, or behind a password.

        There is no reason to block google from indexing a page from
        a normal, real, good, decent, money-making site. Especially
        in the context of what you are saying.

        Google is smart enough to know what to show in serps for
        people who are searching.

        Google has been at this a long, long time, and has the best
        engineers around.

        Nobody in their right search-mind is ever going to see those pages
        in serps, much less click on the dang things, even if they did!

        Paul
        Okay, I say what you're saying. Basically, I'm trying to optimise the site, Demon-clothing.co.uk. It's hardly getting any hits for some reason. My mates, who actually owns the business and I do the website for him this convinced that it's got something to do with eBay, they sell these jackets for a lot less than he can. Anyway, I would like to try and optimise the website if I can.

        If you get a few minutes, check this video out. It's quite interesting and it's where I got the information from regarding blocking some pages. I would appreciate some opinions on the content in this video. It does make very interesting listening, although.sure I would pay the money they were asking.

        Clever Google SEO Traffic Secrets & More... | Affilorama
        {{ DiscussionBoard.errors[8924354].message }}
  • Profile picture of the author nettiapina
    If you really want to politely ask Google to not index some pages either robots.txt or per-page settings are a good way to do it. I would do it only under special circumstances, such as sites search engine leaking results pages to Google.
    Signature
    Links in signature will not help your SEO. Not on this site, and not on any other forum.
    Who told me this? An ex Google web spam engineer.

    What's your excuse?
    {{ DiscussionBoard.errors[8924394].message }}
  • Profile picture of the author yukon
    Banned
    I disagree with the comment that Google is smart enough to weed out pages you don't want in the SERPs.

    Reality is Google will eventually index every single thing hosted on your hosting account If it's a supported filetype unless you block Google from showing specific pages/filetypes.

    If you have a specific page you don't want in the SERPs, you have 3 options:
    • Noindex tag on the specific page
    • Noindex in the robots.txt
    • Noindex in the htaccess file

    Here's your warning...

    Noindex tags will remove a page from the SERPs the next time Googlebot loads the specific page. You better know what your doing because noindexing a web page in Google SERPs is a heck of a lot easier than indexing a web page, get it right the first time (If it's actually necessary).

    I only took a quick look at your site, your on-page SEO is weak.
    {{ DiscussionBoard.errors[8924775].message }}
    • Profile picture of the author Oscarfishlover
      Originally Posted by yukon View Post

      I disagree with the comment that Google is smart enough to weed out pages you don't want in the SERPs.

      Reality is Google will eventually index every single thing hosted on your hosting account If it's a supported filetype unless you block Google from showing specific pages/filetypes.

      If you have a specific page you don't want in the SERPs, you have 3 options:
      • Noindex tag on the specific page
      • Noindex in the robots.txt
      • Noindex in the htaccess file

      Here's your warning...

      Noindex tags will remove a page from the SERPs the next time Googlebot loads the specific page. You better know what your doing because noindexing a web page in Google SERPs is a heck of a lot easier than indexing a web page, get it right the first time (If it's actually necessary).

      I only took a quick look at your site, your on-page SEO is weak.
      Thanks. Yes, I know it's very weak.This is the reason I don't do these silly little online shops any more, they are more trouble than they are worth. Everybody is selling Harrington jacket here in the UK, many people are paying for the adverts so it is virtually impossible to get anywhere near the top organically. I think I'm on the second page with the term "buy Harrington jacket"

      What's quite annoying is a couple of years ago it got absolutely tons of hits each day, in fact we were in the top five for black Harrington jacket, those days have long gone. I'll have to check and make sure I've got a meaningful description on each product. The problem I've got is we sell something like 12 Harrington jacket, they basically use the same description, apart from the colour. It's difficult to write a different description for each jacket, I'm probably getting penalised for using the same description on each one.

      Anyway, thankfully nobody relies on the website to pay their bills. That's Amazon and eBay for you, they can sell all the same stuff for a pittance. We could set up an online store on eBay, but from what my mate said, he would make about £1 from each sale, it just isn't worth the bother when you cannot compete with these massive companies.
      {{ DiscussionBoard.errors[8925162].message }}
  • Profile picture of the author socialmediaandseo
    User-agent: *
    NoAllow:
    Sitemap: siteurl with sitemap.xml

    User-Agent: MSNBot
    NoAllow:

    User-Agent: Slurp
    NoAllow:

    User-Agent: AskJeeves
    NoAllow:

    User-Agent: CoolBot
    NoAllow:

    User-Agent: Digger
    NoAllow:

    User-Agent: FastCrawler
    NoAllow:

    User-Agent: Inktomi Slurp
    NoAllow:

    User-Agent: Mediapartners-Google
    NoAllow:

    User-Agent: Googlebot-Image
    NoAllow
    {{ DiscussionBoard.errors[8929129].message }}

Trending Topics