Robots.txt analysis

  1. How to check the file
  2. How to find out if the robot will crawl a specific URL
  3. How to track file changes
  4. FAQ

The Robots.txt analysis tool helps you to check if the robots.txt file is correct. You can enter the file content, validate it and then copy to robots.txt.

This tool will also help you to track changes to the file and download its specific version.

  1. How to check the file
  2. How to find out if the robot will crawl a specific URL
  3. How to track file changes
  4. FAQ

How to check the file

If the site was added to Yandex.Webmaster and the site management rights were verified

The file content appears on the Tools → Robots.txt analysis page as soon as the rights to manage the website are verified.

If the content is shown on the Robots.txt analysis page, click Check.

If the site has not been added to Yandex.Webmaster
  1. Go to the Robots.txt analysis page.
  2. In the Site to be checked field, enter the address of your website. For example, https://example.com.
  3. Click the icon. The content of robots.txt and the results of the analysis will be shown below.

In the sections intended for the Yandex robot (User-agent: Yandex or User-agent: *), the validator checks the directives using the robots.txt terms of use. The other sections are checked against the standard.

After the check, you may see:
  • Warnings. They report a deviation from the rules that can be corrected by the tool itself. Warnings also indicate a potential problem with typos or inaccuracies in the directives.
  • Errors in the file. This means that the tool cannot process a string, section, or the entire file due to severe syntax errors in the directives.
For more information, see Robots.txt parsing errors.

How to find out if the robot will crawl a specific URL

When your robots.txt file is uploaded to Yandex.Webmaster, the Robots.txt analysis page displays the Check if URLs are allowed block.

  1. In the URL list field, enter the address of the page you want to check. You can specify the URL completely or relatively to the website root directory. For example, https://example.com/page/ or /page/.
  2. Click Check.

If the URL allows indexing by Yandex bots, the icon will appear next to it. Otherwise, the address will be highlighted in red.

How to track file changes

Note. A six-month history of changes is available. The maximum number of saved versions is 100.

To learn about changes to the robots.txt file promptly, please configure notifications.

Yandex.Webmaster regularly checks the file for updates and saves the versions along with the date and time of the change. To view them, go to Tools → Robots.txt analysis.

The list of versions is shown, provided that all of the following conditions are met:
  • You added the website to Yandex.Webmaster and verified your right to manage the website.
  • Yandex.Webmaster keeps the information on changes in robots.txt.
You can:
View current and previous versions of the file

In the robots.txt version list, choose the file version. A field below shows robots.txt along with the parsing results.

Download the selected file version
  1. In the robots.txt version list, choose the file version.
  2. Click the Download button. The file will be saved on your device in TXT format.

FAQ

“This URL does not belong to your domain” error

Most likely, you included a mirror in the list of your site URLs. For example, http://example.com instead of http://www.example.com (technically, these are two different URLs). Technically, these are two different URLs. The URLs in the list must belong to the site for which robots.txt is checked.

Specify the tool in which you found the error, describe the situation in as much detail as possible, and if necessary, attach a screenshot that illustrates it.