The User-agent directive

In the robots.txt file, the bot checks for entries starting with User-agent: and looks for either Yandex substrings (case doesn’t matter) or *. If the string User-agent: Yandex is detected, the string User-agent: * is ignored. If the User-agent: Yandex and User-agent: * strings are not found, the robot is considered to have unlimited access.

You can enter separate directives for the Yandex robots: To learn more about how robots account for them, see the Yandex robots in the server logs section.

If there are directives for a specific robot, directives User-agent: Yahoo and User-agent: * aren't used.

Example:

User-agent: YandexBot # will be used only by the main indexing bot
Disallow: /*id=

User-agent: Yandex # will be used by all Yandex bots
Disallow: /*sid= # except the main indexing bot

User-agent: * # will not be used by Yandex bots
Disallow: /cgi-bin 

According to the standard, you should insert a blank line before every User-agent directive. The # character designates commentary. Everything following this character, up to the first line break, is disregarded. See details in the Processing the # symbol section.