Robots.txt
Article translated by machine
This text is a machine translation. A revised version is planned.
The file robots.txt is a text file that can be stored on the server of a website. It determines whether and which subpages of the website may be visited by search engine crawlers and included in their index.
This allows certain subpages to be excluded from the search result.
Example: You can use the file robots.txt to exclude all archive contents of a website from the search result.
However, some search engine crawlers ignore the robots.txt file. Therefore, subpages that must not appear in search engines should be password protected.