Site indexing by robots

A robot (crawler) keeps a list of URLs it can index and regularly downloads the documents located there. If the robot finds a new link when analyzing the document, it adds the link to the list. Thus, any document or site that has links to it can be found by the robot, and hence by the Yandex search.

What are the types of indexing robots?

Yandex has several types of robots that perform different functions. For example, there is a robot that indexes RSS feeds to search in blogs. Or a robot that indexes only images. The most important one is the main indexing robot that searches and indexes information to form the main search database.

The main robot gets help from the fast robot that quickly indexes fresh and up-to-date information. If you see two copies of the same document among the indexed pages of your site, it is likely that the document was indexed by the fast robot separately from the main robot.

For more information on how to distinguish between Yandex robots in your server logs, go to the help page.