The Web Robots and The Robot Exclusion Standard

The Web Robots

Web Robots also known as Web Wanderers, Crawlers, or Spiders are programs that traverse the Web automatically. Search engines such as Google and Bing a use web spiders, also known as robots, to create the indexes for their search databases.  These robots transverse HTML trees by loading pages and following hyperlinks, and they report the text and/or meta-tag information to create search indexes.  ROBOTS.TXT, a file that spiders look in for information on how the site is to be cataloged.  It is a ASCII text file that sits in the document root of the server.  It defines what documents and/or directories that confirming spiders are forbidden to index.

In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting).
These incidents indicated the need for established mechanisms for WWW servers to indicate to robots which parts of their server should not be accessed. This standard addresses this need with an operational solution.

The Robot Exclusion Standard

The robot exclusion protocol was introduced by Martijn Koster in 1994 to deal with problems that had been arising due to the increasing popularity of the internet and the toll web spiders were having on system resources.  Some of the problems were caused by robots rapid-firing requests, that is loading pages in rapid succession.  Other problems such as robots indexing information deep in directory trees, temporary information, and even accessing cgi-scripts.  The robot exclusion protocol was quickly adopted by webmasters and web robot makers as a way to organize and control the indexing process.

 The method used to exclude robots from a server is to create a file on the server which specifies an access policy for robots. This file must be accessible via HTTP on the local URL "/robots.txt". The contents of this file are specified below.
This approach was chosen because it can be easily implemented on any existing WWW server, and a robot can find the access policy with only a single document retrieval.
A possible drawback of this single-file approach is that only a server administrator can maintain such a list, not the individual document maintainers on the server. This can be resolved by a local process to construct the single file from a number of others, but if, or how, this is done is outside of the scope of this document.
The choice of the URL was motivated by several criteria:
  • The filename should fit in file naming restrictions of all common operating systems.
  • The filename extension should not require extra server configuration.
  • The filename should indicate the purpose of the file and be easy to remember.
  • The likelihood of a clash with existing files should be minimal. 

How to create a /robots.txt file

Where to put it

The short answer: in the top-level directory of your web server.
The longer answer:
When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and puts "/robots.txt" in its place.
For example, for ", it will remove the "/shop/index.html", and replace it with "/robots.txt", and will end up with "".
So, as a web site owner you need to put it in the right place on your web server for that resulting URL to work. Usually that is the same place where you put your web site's main "index.html" welcome page. Where exactly that is, and how to put the file there, depends on your web server software.
Remember to use all lower case for the filename: "robots.txt", not "Robots.TXT.

The Format

The format and semantics of the "/robots.txt" file are as follows: The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form "<field>:<optionalspace><value><optionalspace>". The field name is case insensitive.
Comments can be included in file using UNIX bourne shell conventions: the '#' character is used to indicate that preceding space (if any) and the remainder of the line up to the line termination is discarded. Lines containing only a comment are discarded completely, and therefore do not indicate a record boundary.
The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below. Unrecognised headers are ignored.
The value of this field is the name of the robot the record is describing access policy for. If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record.
The robot should be liberal in interpreting this field. A case insensitive substring match of the name without version information is recommended.
If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.
The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html. Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record.
The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.


The following example "/robots.txt" file specifies that no robots should visit any URL starting with
"/cyberworld/map/" or "/tmp/", or /foo.html:
# robots.txt for
 User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
This example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/", except the robot called "cybermapper":
# robots.txt for
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
This example indicates that no robots should visit
this site further: 
# go away
User-agent: *
Disallow: / 
To Know More Visit:


Popular posts from this blog

Ubuntu 18.04 Nginx PHP 7 MariaDB Server Setup

Sending an Email to Multiple Recipients - Cc: and Bcc: