How to make sure that a robot belongs to Yandex

Some Yandex robots download web documents for purposes other than indexing. To avoid unintentional blocking by site owners, they may ignore the restrictive directives of the file robots.txt intended for arbitrary robots (User-agent: *).

In addition, robots may ignore some robots.txt restrictions for certain sites if there is an agreement between «Yandex» and site owners.

To restrict such robots from accessing the site, use directives meant for them specifically. For example:

User-agent: YandexCalendar
Disallow: /

User-agent: YandexMobileBot
Disallow: /private/*.txt$

Robots use an autonomous network: AS13238 and AS208722, as well as IP addresses that change frequently, so their list isn't disclosed.

When the robot accesses the page, your server logs may display the User-agent and version of the browser used for crawling the site. For example, Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268.

* The combination of W.X.Y.Z characters is a placeholder for the Chrome browser version used by the user agent. For example: 41.0.2272.96.

How do I protect myself from fake robots that pretend to be Yandex robots?

To protect yourself from fake robots, use the reverse DNS lookup filter, as described above. We recommend this method instead of managing access by IP addresses, as it is more resistant to changes in the Yandex internal networks.

There's too much traffic going back and forth between my web server and your robot. Does Yandex support downloading of compressed pages?

Yes, it does. Each time the Yandex robot requests a page, it says: “Accept-Encoding: gzip,deflate” . This means you can set up your web server to reduce the traffic between the server and our robot. However, note that sending compressed content increases CPU usage on your server. If it's overloaded, it can cause problems. To download gzip and deflate, the robot applies the rfc2616 standard, section 3.5 .

Contact support