Robots.txt analysis

The Robots.txt analysis tool helps you to check if the robots.txt file is correct or enter the file text, validate it and copy to robots.txt.

This tool will also help you to track changes to the file and download its specific version.

How to check the file

If the site was added to Yandex.Webmaster and the site management rights were verified

The file contents will appear on the Tools → Robots.txt analysis page as soon as the rights to manage the website are verified.

If the content is shown on the Robots.txt analysis page, click Check.

If the site has not been added to Yandex Webmaster

1. Go to the Robots.txt analysis page.
2. In the Site to be checked field, enter the address of your website. For example, https://example.com.
3. Click the icon. The content of robots.txt and the results of the analysis will be shown below.

In the sections intended for the Yandex robot (User-agent: Yandex or User-agent: *), the validator checks the directives using the robots.txt terms of use. The other sections are checked against the standard.

After the check, you may see:

• Warnings. They report a deviation from the rules that can be corrected by the tool itself. Warnings also indicate a potential problem with typos or inaccuracies in the directives.
• Errors in the file. This means that the tool cannot process a string, section, or the entire file due to severe syntax errors in the directives.

For more information, see Robots.txt parsing errors.

How to find out if the robot will crawl a specific URL

When your robots.txt file is uploaded to Yandex.Webmaster, the Robots.txt analysis page displays the Check if URLs are allowed block.

1. In the URL list field, enter the address of the page you want to check. You can specify the URL completely or relatively to the website root directory. For example, https://example.com/page/ or /page/.
2. Click the Check button.

If the URL allows indexing by Yandex bots, the icon will appear next to it. Otherwise, the address will be highlighted in red.

How to track file changes

To learn about changes to the robots.txt file promptly, please configure notifications.

Yandex Webmaster regularly checks the file for updates and saves the versions along with the date and time of the change. To view them, go to the Tools → Robots.txt analysis page.

The list of versions is shown, provided that all of the following conditions are met:

• You added the website to Yandex Webmaster and verified your right to manage the website.
• Yandex.Webmaster keeps the information on changes in robots.txt.

You can:

View current and previous versions of the file

In the robots.txt version list, choose the file version. A field below shows robots.txt along with the parsing results.

Download the selected file version

1. In the robots.txt version list, choose the file version.
2. Click the Download button. The file will be saved on your device in TXT format.

Questions and answers

An error: “This URL does not belong to your domain”

Most likely, you specified a secondary mirror in the URL list, such as http://example.com instead of http://www.example.com. Technically, these are two different URLs. The URLs in the list must belong to the site for which robots.txt is checked.