Robots.txt analysis error manual


The list of errors that may be encountered while parsing robots.txt.

Error Yandex extension? Description
A rule begins with a character that is neither '/' nor a '*'. Yes

A rule can begin with '/' or a '*' only.

Several matching rules found for User-agent: *. No

Only one rule of this type is allowed.

Several Host directives discovered. Yes

Only one Host directive is allowed.

The robots.txt file is too long. Yes

The number of rules in the file exceeds 1024.

A rule is not preceded by User-agent directive. No

A rule must always be preceded by the User-agent directive. It is possible that an empty line break follows the User-agent directive.

A rule is too long. Yes

The rule exceeds acceptable length (1024 characters).

Incorrect name of the main mirror of the site. Yes

The name of the main mirror of the site in the Host directive contains a syntax error.

Badly formed URL of the Sitemap file. Yes

URL of the Sitemap file must be specified as a complete URL, including http://. E.g.,

Wrong format of Crawl-delay directive. Yes

Wrong time specified in the Crawl-delay directive.

Several Crawl-delay directives have been discovered. Yes

Only one Crawl-delay directive is allowed.

Wrong format of Clean-param directive. Yes

The Clean-param directive specifies one or more parameters that the robot will disregard, as well as the path prefix. The parameters are delimited with '&' and separated from the path prefix by a space.


The list of warnings that may be generated while parsing robots.txt.

Warning Yandex extension? Description
An invalid character may have been used. Yes

A special character different from "*" and "$" was discovered.

Unknown directive was discovered. Yes

A directive not described in the rules for using robots.txt was discovered. It is possible that this directive is used by robots that belong to other search engines.

A syntax error. Yes

A string cannot be interpreted as a robots.txt directive.

Unknown error. Yes

Analysis of robots.txt discovered an unknown error. Contact technical support.

URL validation errors

The list of URL validation errors that may be encountered while parsing robots.txt.

Error Description
A syntax error.

URL syntax error.

This URL does not belong to your domain

The specified URL does not belong to the site for which the robots.txt file is analyzed. You may have specified a URL for one of your site mirrors or misspelled the domain name.