The User-agent directive
In the robots.txt file, the robot checks for records starting with User-agent: and looks for either the substring Yandex (the case doesn't matter) or *. If the User-agent: Yandex string is detected, the User-agent: * string is ignored. If the User-agent: Yandex and User-agent: * strings are not found, the robot is considered to have unlimited access.
You can enter separate directives for the Yandex robots: To learn more about how robots account for them, see the Yandex robots in the server logs section.
If there are directives for a specific robot, directives
User-agent: Yahoo and
User-agent: * aren't used.
User-agent: YandexBot # will only be used by the main indexing bot Disallow: /*id= User-agent: Yandex # will be used by all Yandex bots Disallow: /*sid= # except for the main indexing bot User-agent: * # won't be used by Yandex bots Disallow: /cgi-bin
According to the standard, you should insert a blank line before every User-agent directive. The # character designates commentary. Everything following this character, up to the first line break, is disregarded.