7 {{ upvoteCount | shortNum }}
7 {{ upvoteCount | shortNum }}

Robots.txt file Disallow: /*? Used to?

by neha12
6 replies
  • SEO
    • |
Hi Warriors,

I am worried about this Disallow: /*? code.

Can you all please explain, what is the use of this code in robots.txt file? and what it does?
#disallow #file #robotstxt
  • Profile picture of the author paulgl
    It blocks google from getting to any url that
    has a question mark in it.

    If you have important urls with "?" in them, it's
    blocking them. Chances are, urls with "?" in them
    are useless for search queries. Note I said, "chances
    are," and not "all the time." The wf uses ? for writing
    replies, among other things. You certainly don't want
    those things indexed, although the chance is slight
    anyway. A robots.txt is really not even needed.

    Many forums and other sites have session ids...these
    use "?" as well.

    Paul
    Signature

    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[8407675].message }}
    • Profile picture of the author neha12
      Originally Posted by paulgl View Post

      It blocks google from getting to any url that
      has a question mark in it.

      If you have important urls with "?" in them, it's
      blocking them. Chances are, urls with "?" in them
      are useless for search queries. Note I said, "chances
      are," and not "all the time." The wf uses ? for writing
      replies, among other things. You certainly don't want
      those things indexed, although the chance is slight
      anyway. A robots.txt is really not even needed.

      Many forums and other sites have session ids...these
      use "?" as well.

      Paul
      Paul, first of all thanks for an useful response.

      Can we say it disallow all the links generated dynamically?
      {{ DiscussionBoard.errors[8458109].message }}
  • Profile picture of the author bapparabi
    Originally Posted by neha12 View Post

    Hi Warriors,

    I am worried about this Disallow: /*? code.

    Can you all please explain, what is the use of this code in robots.txt file? and what it does?
    Please refer this link you will find all information what is it and how its done ...

    The Web Robots Pages
    Signature
    Earn $ 1000+ a Month Easily & Automatically - 100% Automated WordPress Hotel, Flight, Cruises, Rental Cars & Amazon Store Search Engine Script
    {{ DiscussionBoard.errors[8407698].message }}
  • Profile picture of the author Arshaan
    Robots TXT disallows Search Engines if needed, you can tell the Search Engines "robots" to NOT index and crawl certain parts of a website or your whole website if needed.

    Many webmasters use this disallow feature so that when their competitors use software's such as SEOmoz or Majestic SEO to analyze their back-link profile they can actually stop this inquiry and keep their back-links private which gives them a huge advantage.
    {{ DiscussionBoard.errors[8408206].message }}
  • Profile picture of the author Clint Faber
    Your robots.txt register requests your net spiders not crawl specific data or directories of the websites. This file ought to be situated on root of your websites, particularly http://ursite.com/robots.txt.

    Robots.txt Disallow

    De disallow req. in a robots.txt file looks as follows:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /shopping-cart.aspx

    Look at Wikipedia.org robots.txt they blocked spiders an have comments that describe why de bots are blocked. Right

    Hope it helps
    {{ DiscussionBoard.errors[8408890].message }}
  • Profile picture of the author khawajahassan
    If you do not want your some urls, you need to create robots.txt.Disallow: /*? mean the url contain '?' spider leave url.
    {{ DiscussionBoard.errors[8458195].message }}

Trending Topics