---
metadata:
- name: generator
content: Diplodoc Platform v5.50.6
alternate:
- https://yandex.com/support/webmaster/en/controlling-robot/robots-txt.md
- https://yandex.com/support/webmaster/tr/controlling-robot/robots-txt.md
---
> **Documentation Index:** Fetch the complete configuration index at https://yandex.com/support/webmaster/en/llms.txt
# Using robots.txt
_Robots.txt_ is a text file that contains site indexing parameters for the search engine robots. In _robots.txt_, you can restrict the indexing of website pages by bots, which can reduce the load on the site and speed up its performance.
{% note info %}
Pages restricted in _robots.txt_ can participate in Yandex search. To remove pages from search, specify the _noindex_ directive in the HTML code of the page or configure the HTTP header. Do not restrict such pages in _robots.txt_, or the Yandex bot can’t index them and detect your instructions. See details in the [How to exclude pages from search](https://yandex.com/support/webmaster/en/yandex-indexing/removing-from-index.md) section.
{% endnote %}
Yandex supports the [Robots Exclusion Protocol](http://www.robotstxt.org/robotstxt.html) with advanced features.
## Requirements to the robots.txt file {#requirement}
Yandex robots correctly process `robots.txt`, if:
- The file size doesn't exceed 500 KB.
- It is a TXT file named "robots", `robots.txt`.
- The file is located in the root directory of the site.
- The file can be accessed by indexing bots:
- The server hosting the site returns an HTTP 200 OK status code.
- The `robots.txt` file redirects to another `robots.txt` file for which the server returns a 200 OK code.
To learn more, see [Server response check](https://webmaster.yandex.com/tools/server-response/).
Yandex supports redirection from the `robots.txt` file located on one site to the file located on another site. In this case, the directives in the target file are taken into account. This redirect can be useful when [moving the site](https://yandex.com/support/webmaster/en/yandex-indexing/moving-site.md).
If the file doesn't meet the requirements, the site is considered open for indexing.
## Recommendations on the content of the file {#recommend}
Yandex supports the following directives:
#|
||
**Directive**
|
**What it does**
||
||
[User‑agent](https://yandex.com/support/webmaster/en/robot-workings/user-agent.md) *
|
Indicates the robot to which the rules listed in `robots.txt` apply.
||
||
[Disallow](https://yandex.com/support/webmaster/en/robot-workings/allow-disallow.md)
|
Prohibits crawling of sections or individual pages of the site.
||
||
[Sitemap](https://yandex.com/support/webmaster/en/robot-workings/sitemap.md)
|
Specifies the path to the `Sitemap` file that is posted on the site.
||
||
[Clean-param](https://yandex.com/support/webmaster/en/robot-workings/clean-param.md)
|
Indicates to the robot that the page URL contains parameters (like UTM tags) that should be ignored when indexing it.
||
||
[Allow](https://yandex.com/support/webmaster/en/robot-workings/allow-disallow.md)
|
Allows indexing site sections or individual pages.
||
|#
\* Mandatory directive.
You'll most often need the Disallow, Sitemap, and Clean-param directives. For example:
```no-highlight
User-agent: * #indicates which bots the directives are set for
Disallow: /bin/ # prohibits links from the “Shopping Cart”.
Disallow: /search/ # prohibits links from the site’s built-in search
Disallow: /admin/ # prohibits links from the admin panel
Sitemap: http://example.com/sitemap # points the bot to the Sitemap file for the site
Clean-param: ref /some_dir/get_book.pl
```
Robots from other search engines and services may interpret the directives in a different way.
{% note info %}
The robot takes into account the case of substrings (file name or path, robot name) and ignores the case in the names of directives.
{% endnote %}
## Using Cyrillic characters {#cyrillic-urls}
The use of the Cyrillic alphabet is not allowed in the `robots.txt` file and server HTTP headers.
For domain names, use [Punycode](https://en.wikipedia.org/wiki/Punycode). For page addresses, use the same encoding as that of the current site structure.
Example of the `robots.txt` file:
* Incorrect:
```no-highlight translate=no
User-agent: Yandex
Disallow: /корзина
Sitemap: сайт.рф/sitemap.xml
```
* Correct:
```no-highlight translate=no
User-agent: Yandex
Disallow: /%D0%BA%D0%BE%D1%80%D0%B7%D0%B8%D0%BD%D0%B0
Sitemap: http://xn--80aswg.xn--p1ai/sitemap.xml
```
## How to set up robots.txt {#how-to-create}
1. In the text editor, create a file named `robots.txt` and add the [directives you need](#recommend) in it.
1. [Check the file](https://yandex.com/support/webmaster/en/indexing-options/robots-txt-analyzer.md) in Yandex Webmaster.
1. Place the file to your site's root directory.
[Sample file](https://download.cdn.yandex.net/from/yandex.ru/support/ru/webmaster/files/robots-txt-example.txt). This file allows indexing of the entire site for all search engines.
## Questions and answers {#qanda}
{% cut "The "Server responds with redirect to /robots.txt request" error occurs on the "Site diagnostics" page in Yandex Webmaster" %}
To ensure Yandex's indexing bot obeys the [robots.txt](https://yandex.com/support/webmaster/en/indexing-options/robots-txt-analyzer.md#robots-txt-analyzer) file, it should be located in your site's root directory and return an HTTP 200 code. If the `robots.txt` file redirects to another `robots.txt` file (for example, when you migrate your site), Yandex obeys the target file.
For the directives in your `robots.txt` file to work as intended, make sure that the file contains correct information. To test the file, add the target site in Yandex Webmaster and verify your site ownership.
To check availability of the `robots.txt` file for the robot, [check the server response](https://webmaster.yandex.com/tools/server-response/).
{% endcut %}
{% cut "Contact support" %}