4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

help with robots.txt file

by ip3
3 replies
  • SEO
    • |
i want to make sure that google and other serch engines can crawl my site, i wanted to know if this txt is allowing them to crawl or not?


User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
#file #robotstxt
  • Profile picture of the author Tim3
    Originally Posted by ip3 View Post

    i want to make sure that google and other serch engines can crawl my site, i wanted to know if this txt is allowing them to crawl or not?


    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search

    Yes, they are crawling its the forward slash ( / ) that blocks the crawl, thus:

    User-agent: Mediapartners-Google
    Disallow: / now blocked



    Edit:

    Do not remove this:

    User-agent: *
    Disallow: /search
    Signature

    {{ DiscussionBoard.errors[3719480].message }}
  • Profile picture of the author malco714
    my site went from page one to 44 and no meta disc. and goes to a page called stuff gate dont know why
    {{ DiscussionBoard.errors[4709221].message }}
  • Profile picture of the author rakesh1600193
    yes you have the right code.

    you can check it in google webmasters tools also.
    {{ DiscussionBoard.errors[4709342].message }}

Trending Topics