Webmaster
How Yandex indexes sites
A site on search results page

How to check that a robot belongs to Yandex

Some robots can disguise themselves as Yandex robots by indicating the corresponding User Agent. You can check the authenticity of a robot using reverse DNS lookup.

Just follow these steps:

  • Determine the IP address of the user-agent in question using your server logs. All Yandex robots are represented by a set User agent.

  • Use a reverse DNS lookup of the received IP address to determine the host domain name.

  • After determining the host name, you can check whether or not it belongs to Yandex. All Yandex robots have names ending in 'yandex.ru','yandex.net' or 'yandex.com'. If the host name has a different ending, the robot does not belong to Yandex.

  • Finally, make sure that the name is correct. Use a forward DNS lookup to get the IP address corresponding to the host name. It should match the IP address used in the reverse DNS lookup. If the IP addresses do not match it means that the host name is fake.

Yandex robots in server logs

Yandex has many robots that take different forms:

  • Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) — The main indexing robot.

  • Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots) — Indexing robot.
  • Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots) — Downloads pages to check user accessibility, interprets robots.txt in a special way.
  • Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots) — Determines if webpage layouts are suitable for mobile devices, interprets robots.txt in a special way.
  • Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots— Generates dynamic banners, interprets robots.txt in a special way.
  • Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 (compatible; YandexScreenshotBot/3.0; +http://yandex.com/bots) — Takes screenshots of the page, interprets robots.txt in a special way.
  • Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) — The Yandex.Images indexer.

  • Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots) — The Yandex.Video indexer.

  • Mozilla/5.0 (compatible; YandexVideoParser/1.0; +http://yandex.com/bots) — The Yandex.Video indexer, interprets robots.txt in a special way.
  • Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots) — Indexes multimedia content.

  • Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots) — The blog search robot that indexes comments on posts.

  • Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots) — Indexes site favicons.

  • Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots) — Robot for the Yandex.Webmaster service.

  • Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots) — Accesses the page for validating the microformat when using the Microformat validator.

  • Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots) — Robot for mobile services.

  • Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots) — The “fetcher” for Yandex.Direct that checks if ad links are correct before sending them for review, interprets robots.txt in a special way.

  • Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots) — Yandex.Calendar robot used for syncing with other calendars, interprets robots.txt in a special way.

  • Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots) — The sitelinks “fetcher” used for checking the availability of webpages determined to be sitelinks.

  • Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots) — The Yandex.Metrica robot, interprets robots.txt in a special way.

  • Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots) — Antivirus robot that checks pages for malware.

  • Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots) — Vertical search robot.

  • Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots) — Determines if a site is a mirror.

There are many IP addresses that Yandex robots can “originate” from, and these addresses change frequently. We are therefore unable to offer a list of IP addresses and we do not recommend using a filter based on IP addresses.

Rate this article
Thank you for your feedback!