Robots.txt analysis

How to check the file

If the website has been added to Yandex.Webmaster and the website management rights have been verified

The file content appears on the Tools → Robots.txt analysis page as soon as the rights to manage the website are verified.

If the content is shown on the Robots.txt analysis page, click Check.

If the site has not been added to Yandex.Webmaster
  1. Go to the Robots.txt analysis page.
  2. In the Site to be checked field, enter the address of your website. For example, https://example.com.
  3. Click the icon. The content of robots.txt and the results of the analysis will be shown below.

In the sections intended for the Yandex robot (User-agent: Yandex or User-agent: *), the validator checks the directives using the robots.txt terms of use. The other sections are checked against the standard.

After the check, you may see:
  • Warnings. They report a deviation from the rules that can be corrected by the tool itself or indicate a possible typo or incorrect directive.
  • Errors in the file. This means that the tool cannot process a string, section, or the entire file due to severe syntax errors in the directives.
For more information, see Robots.txt parsing errors.

How to find out if the robot will crawl a specific URL

When your robots.txt file is uploaded to Yandex.Webmaster, the Robots.txt analysis page displays the Check if URLs are allowed block.

  1. In the URL list field, enter the address of the page you want to check. You can specify the URL completely or relatively to the website root directory. For example, https://example.com/page/ or /page/.
  2. Click the Check button.

If the URL allows indexing by Yandex bots, the icon will appear next to it. Otherwise, the address will be highlighted in red.

How to track file changes

Note. A six-month history of changes is available. The maximum number of saved versions is 100.

To learn about changes to the robots.txt file promptly, please configure notifications.

Yandex.Webmaster regularly checks the file for updates and saves the versions along with the date and time of the change. To view them, go to Tools → Robots.txt analysis.

The list of versions is shown, provided that all of the following conditions are met:
  • You added the website to Yandex.Webmaster and verified your right to manage the website.
  • Yandex.Webmaster keeps the information on changes in robots.txt.
You can:
View current and previous versions of the file

In the robots.txt version list, choose the file version. A field below shows robots.txt along with the parsing results.

Download the selected file version
  1. In the robots.txt version list, choose the file version.
  2. Click the Download button. The file will be saved on your device in the TXT format.

Questions and answers

“This URL does not belong to your domain” error

Most likely, you included a mirror in the list of your site URLs. For example, http://example.com instead of http://www.example.com (technically, these are two different URLs). The URLs in the list must belong to the site for which robots.txt is checked.