---
metadata:
  - name: generator
    content: Diplodoc Platform v5.55.2
alternate:
  - https://yandex.com/support/webmaster/en/indexing-options/robots-txt-analyzer.md
  - https://yandex.com/support/webmaster/tr/indexing-options/robots-txt-analyzer.md
  - href: en/indexing-options/robots-txt-analyzer.md
    type: text/markdown
    title: Markdown version
  - href: ../llms.txt
    type: text/markdown
    title: llms.txt
---
> **Documentation Index:** Fetch the complete configuration index at https://yandex.com/support/webmaster/en/llms.txt


# Robots.txt analysis

The **Robots.txt analysis** tool helps you to check if the [robots.txt](https://yandex.com/support/webmaster/en/controlling-robot/robots-txt.md) file is correct. You can enter the file content, validate it and then copy to `robots.txt`.

This tool will also help you to [track changes to the file and download its specific version](#changelog).


## How to check the file {#check}

**If the site was added to Yandex Webmaster and the site management rights were verified**

 
:   The file contents appear on the **Tools** → **Robots.txt analysis** page as soon as the site management rights are verified.
    
    If the content is shown on the **Robots.txt analysis** page, click **Check**.

**If the site hasn't been added to Yandex Webmaster**

 
:   1. Go to the [Robots.txt analysis](https://webmaster.yandex.com/tools/robotstxt/) page.
    1. In the **Site to be checked** field, enter the address of your website. For example, https://example.com.
    1. Click the ![](../_assets/check-robots-download.svg) icon. The content of `robots.txt` and the results of the analysis will be shown below.

In the sections intended for the Yandex robot (User-agent: Yandex or User-agent: *), the validator checks the directives using the [robots.txt terms of use](https://yandex.com/support/webmaster/en/controlling-robot/robots-txt.md). The other sections are checked against the [standard](http://www.robotstxt.org/robotstxt.html).

After the check, you may see:
- Warnings. They report a deviation from the rules that can be corrected by the tool itself. Warnings also indicate a potential problem with typos or inaccuracies in the directives.
- Errors in the file. This means that the tool cannot process a string, section, or the entire file due to severe syntax errors in the directives.

For more information, see [Robots.txt parsing errors](https://yandex.com/support/webmaster/en/error-dictionary/robots-txt.md).


## How to find out if the robot will crawl a specific URL {#list}

When your `robots.txt` file is uploaded to Yandex Webmaster, the **Robots.txt analysis** page displays a **Check if URLs are allowed** section.

1. In the **URL list**  field, enter the address of the page you want to check. You can specify the URL completely or relatively to the website root directory. For example, https://example.com/page/ or /page/.
1. Click the **Check** button.

If a URL allows indexing by Yandex bots, the ![](../_assets/check-robots-allow.svg) icon will appear next to it. Otherwise, the address will be highlighted in red.


## How to track file changes {#changelog}


{% note info %}

A six-month history of changes is available. The maximum number of saved versions is 100.

{% endnote %}


To learn about changes to the `robots.txt` file promptly, please [configure notifications](https://yandex.com/support/webmaster/en/service/notifications.md#sites).

Yandex Webmaster regularly checks the file for updates and saves the versions along with the date and time of the change. To view them, go to **Tools** → **Robots.txt analysis**.

The list of versions is shown, provided that all of the following conditions are met:
- You added the site to Yandex Webmaster and verified your site management rights.
- Yandex Webmaster has information on changes in `robots.txt`.

You can:

**View current and previous versions of the file**

 
:   In the **robots.txt version** list, choose the file version. A field below shows `robots.txt` along with the parsing results.

**Download the selected file version**

1. In the **robots.txt version** list, choose the file version.
1. Click the **Download** button. The file will be saved on your device in TXT format.


## Questions and answers {#qanda}


{% cut "An error: “This URL does not belong to your domain”" %}


Most likely, you specified an alternate address in the URL list, such as `http://example.com` instead of `http://www.example.com`. Technically, these are two different URLs. The URLs in the list must belong to the site for which `robots.txt` is checked.


{% endcut %}


<div class="cut-button">


{% cut "Report an error in the tool" %}

Specify the tool in which you found the error, describe the situation in as much detail as possible, and if necessary, attach a screenshot that illustrates it.





<div style="padding: 15px;
        margin: 10px 0;
        background: #FFFFFF;
        border-radius: 10px;
        border: 1px solid var(--yc-color-line-generic);">
    <iframe style="background: #FFFFFF;"
        height="700"
        width="100%"
        frameborder="0"
        src="https://forms.yandex.com/surveys/3522/?&theme1=Oshibka v rabote instrumenta&iframe=1&lang=en">
    </iframe>
</div>






{% endcut %}


</div>


<!-- source: en/_includes/cut-button-style.md -->

<!-- endsource: en/_includes/cut-button-style.md -->

<!-- source: en/_includes/table-style.md -->

<!-- endsource: en/_includes/table-style.md -->

<!-- source: en/_includes/footer.md -->
<div width="100%" align="center" class="main-footer"> 
<br><br><br>
You can also go to 
<br><br>




[![](../_assets/logo/logo-pic.svg)](https://webmaster.yandex.com/) [![](../_assets/logo/logo-en.svg)](https://webmaster.yandex.com/)



</div>


<!-- source: en/_includes/styles/footer-style.md -->

<!-- endsource: en/_includes/styles/footer-style.md -->
<!-- endsource: en/_includes/footer.md -->

<!-- source: en/_includes/dl-indentation.md -->

<!-- endsource: en/_includes/dl-indentation.md -->