What it is, If my Robots.txt file showing this coding ?

by GauravSeo1388

Posted: 9 years ago 4 replies

INTERNET MARKETING

hello warriors,

I want to confirm that i am working on one of the website , suppose http://www.xyz.com/,
and when i am opening its robots.txt file . It is showing this type of coding as mentioned below in bold words, Kindly look on bold words:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Whats these files mean, Is it right to disallow these files to be crawled by search engines. I have no exact idea about it.

Kindly put your knowledgeable ideas regarding my query.

#coding #file #robotstxt #showing

nizamkhan 9 years ago

This resource will be useful to you, check it out:

The Web Robots Pages

- Nizam
- [ 1 ] Thanks
- 1 reply
Signature
[FREE DOWNLOAD] My list of 101 FREE Tools and Services for Online Marketers!
{{ DiscussionBoard.errors[9934117].message }}
- GauravSeo1388 9 years ago
  
  Originally Posted by nizamkhan
  
  This resource will be useful to you, check it out:
  
  The Web Robots Pages
  
  - Nizam
  
  Hi Nizam,
  
  Thanks for your response. But i am aware about the coding of robots.txt.
  
  what my question is, I wnt to know about the files name that i have mentioned below again in bold words :
  
  User-agent: *
  Disallow: /wp-admin/
  Disallow: /wp-includes/
  
  What does these files name means, they are prevented from crawlers, Is it okey to prevent from crawling or Should we have to make it crawled by search engine crawlers?
  
  Thanks
  
  {{ DiscussionBoard.errors[9934145].message }}
Biswajit Chopdar 9 years ago

It's robot.txt file which says the crawlers of the search engines like google not to crawl this directories.

The /wp-admin/ and the /wp-includes/ contains your wordpress admin files.

And it's right to stop search engines crawl these files. Else these files may appear in search results.

-Biswajit
- Thanks
{{ DiscussionBoard.errors[9934130].message }}
FunMakingMoney 9 years ago

If I don't want the se to crawl my page I would just use the setting inside wp. I have never play with this robot before.
- Thanks
Signature
Stop wasting money! Success is at your fingertips! If you plan to make $1200 to $10,000/mth. It only cost you $10.
{{ DiscussionBoard.errors[9934143].message }}

What it is, If my Robots.txt file showing this coding ?

Trending Topics

Newbie

How did you turn it all around?

A healthy fascination for Hollywood...

Any thoughts on golf's latest phenom...Scottie Scheffler??

best high quality crypto traffic.