The User-agent directive
In the robots.txt file, the robot checks for records starting with User-agent: and looks for either the substring Yandex (the case doesn't matter) or *. If the User-agent: Yandex string is detected, the User-agent: * string is ignored. If the User-agent: Yandex and User-agent: * strings are not found, the robot is considered to have unlimited access.
You can enter separate directives for the Yandex robots: To learn more about how robots account for them, see the Yandex robots in the server logs section.
If there are directives for a specific robot, directives
User-agent: Yahoo and
User-agent: * aren't used.
User-agent: YandexBot # will be used only by the main indexing robot Disallow: /*id= User-agent: Yandex # will be used by all Yandex robots Disallow: /*sid= # other than the main indexing User-agent: * # won't be used by Yandex robots Disallow: /cgi-bin
According to the standard, you should insert a blank line before every User-agent directive. The # character designates commentary. Everything following this character, up to the first line break, is disregarded.