#robotstxt

  • 18 {{ upvoteCount | shortNum }}

    Blocking bots with htaccss and robots.txt files

    Shaun J in SEO

    Hi Does anyone know why you should use both the htaccess file and robots.txt to block certain bots from accessing your site instead of just using the robots.txt? I thought ... [read more]

  • 10 {{ upvoteCount | shortNum }}

    Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay to proceed with?

    michealtech in SEO

    Hi all, We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is ... [read more]

  • 4 {{ upvoteCount | shortNum }}

    Robots.txt & internal filter links blocked

    DigitalKnowledge in SEO

    Our internal links appear to be blocked by robots.txt when they are permutated beyond category or brand pages. We did this so that Google wouldn't index all the filters we ... [read more]

  • 6 {{ upvoteCount | shortNum }}

    Which is the best option Meta Robot tags or robots.txt?

    stephenrobes in SEO

    Hi friends help me to know Which is the best option Meta Robot tags or robots.txt?

  • 8 {{ upvoteCount | shortNum }}

    robots.txt vs meta

    Voora in SEO

    Robots.txt Vs Meta Robots.Which one is better for Effective SEO?

  • 3 {{ upvoteCount | shortNum }}

    Help in Robots.txt file !!!

    benthomas642 in SEO

    Hi there, I need your help. I m confused after looking this in 1 website: Disallow: *? What does it mean? Can anyone please explain here.

  • 2 {{ upvoteCount | shortNum }}

    What is disallow in robots.txt file?

    avemfly619619 in SEO

    Hlo Friends, What is disallow in robots.txt file?

  • 3 {{ upvoteCount | shortNum }}

    Search Console can't fetch Robots.txt !

    TheOptimizer in SEO

    Hello everyone, I'm having an issue I've not had before with google search console. https://www.allianzemeklilik.com.tr/robots.txt When I check the robots.txt file it says: "robots.txt fetch failed You have a robots.txt ... [read more]

  • 7 {{ upvoteCount | shortNum }}

    How important is robots.txt for SEO

    diff122 in

    Hi Warriors, Just need some suggestions regarding the importance of robots.txt for good SEO. How important is that? The reason why I am asking is I have been ranking on ... [read more]