FAQ
   manual.pipni.com > Web  |  Search  |  Problems?

What is the file robots.txt good for? How shall I prevent robots from accessing certain portions of my web?
If you wish to prevent some robots to access a certain portion of your web, create the file robots.txt in the /www directory. The file might look like this: 
User-agent: *
Disallow: /dir1/subdir
Disallow: /dir2
Disallow: /dir3
The directive User-agent defines the types of indexing robots affected (an asterisk stands for "any").
The directive Disallow specifies which directories shouldn't be indexed by robots.
Note: Not every robot "obeys" the file robots.txt.
Total count of tips: 141
Created by PIPNI
Powered by Firebird