What is robots.txt file? Where it is used?
What is the purpose for robots.txt file?
Any of our Web Site (http://www.qualitypointtech.net) used this techniques?

In Search I found Following details.
If a site owner wishes to give instructions to web robots they must place a text file called robots.txt in the root of the web site hierarchy (e.g. http://www.qualitypointtech.com/robots.txt). This text file should contain the instructions in a specific format . Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. If this file doesn't exist web robots assume that the web owner wishes to provide no specific instructions.
Thanks
http://en.wikipedia.org/wiki/Robots_exclusion_standard

Anybody known more details About robots.txt file just explain here.

Share |

Following Post explains what is robots.txt
http://www.bestarticleinformationblog.i ... -txt-file/

Share |

Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from XnYnZ.com

A robots.txt is a simple text file that can be created with Notepad. each robot.txt file should contain two parts
1. User agent: * means all search bots should use the instruction to crawl the website. unless your site is a complex one you need not seperate instructions for seperate spiders.
2.disallow : contains the files which should be avoided from crawling.
Eg.
User-agent: *
Disallow: /wp-

Share |

basically, robots.txt file used to tell search engine robots which pages do the admin want to crawl and which one don't...... Like, some pages admin want are okay if they will not crawl and come in search engine rankings .....
A simple robots.txt file is:-

Syntax :

User-agent: *
Disallow: /

Share |