The User-agent directive

In the robots.txt file, the robot checks for records starting with User-agent: and looks for either the substring Yandex (the case doesn't matter) or *. If the User-agent: Yandex string is detected, the User-agent: * string is ignored. If the User-agent: Yandex and User-agent: * strings are not found, the robot is considered to have unlimited access.

You can enter separate directives for the Yandex robots: To learn more about how robots account for them, see the Yandex robots in the server logs section.

If there are directives for a specific robot, directives User-agent: Yahoo and User-agent: * aren't used.

Example:

User-agent: YandexBot # will only be used by the main indexing bot
Disallow: /*id=

User-agent: Yandex # will be used by all Yandex bots
Disallow: /*sid= # except for the main indexing bot

User-agent: * # won't be used by Yandex bots
Disallow: /cgi-bin 

According to the standard, you should insert a blank line before every User-agent directive. The # character designates commentary. Everything following this character, up to the first line break, is disregarded.