How to control the robot request rate to my site

If your server is overloaded and cannot keep up with our robot download requests, you should use the Crawl-delay directive. It will allow you to specify the minimum amount of time (in seconds) between our search robot downloading one page and starting the next. To ensure compatibility with robots that don't fully comply with the robots.txt processing standard, the Crawl-delay directive will need to be added to the group beginning with User-agent, and immediately following the Disallow (Allow) directive.

The Yandex search robot supports decimal values of Crawl-delay, for example, 0.5. This doesn't mean that our robot will visit your site every 0.5 seconds, but gives our robot more freedom, and improves the indexing of your site.


User-agent: Yandex
Crawl-delay: 2 # specifies a 2 second timeout
User-agent: *
Disallow: /search
Crawl-delay: 4.5 # specifies a 4.5 second timeout