Checking the server's response

You can use the “Server response check” tool to find out whether the site pages are accessible to the Yandex robots.

Attention. If the server response differs from the HTTP status code 200 OK or it returns a message from the list below, the page is excluded from the search results.

To check the server's response, go to the Server response check page and follow these steps:

  1. In the Requested URL field, enter the URL of the page you want to check. For example, https://example.com.
  2. In the Robot field, select the robot to check if it can access the page. The drop-down list contains the main robots that crawl most sites. Once you select the robot, its User-agent is displayed under the field. For more information, see Yandex robots in server logs.
  3. Click Check.
You can also check the response if your server responds to the If-Modified-Since header with the HTTP status code 304 (Not Modified). To do this, specify the date and time in the If-Modified-Since field:
  • When the page being checked was last updated. In this case, the received response must contain the HTTP status code 200 OK.
  • A different date: you'll get a response with code 304 (Not Modified).

After the check is completed, Yandex.Webmaster will display the server's response to the robot. For more information, see HTTP status codes reference.

Note. A response received using the “Server response check” tool may differ from the actual response to the robot, since the tool has a different IP. You can check the response in your server logs.

In addition to the response, the following messages may be displayed:

Message Note
Document failed to load It appears for the following reasons:
  • The robot's request is blocked on the server side or a connection fault occurs when trying to get a response. In this case, check whether the robot is blocked by IP.
  • DNS error. Make sure that the site is accessible and the DNS is configured properly. If the settings have been changed recently, the data is updated within 24 hours.
  • Response timeout expired. A response should be received within 3 seconds.
Failed to connect to the server
A connection failure occurred
The noindex tag disables URL indexing The page has the robots meta tag with the noindex directive.
URL indexing is disallowed in the robots.txt file The robots.txt file contains the Disallow directive that prohibits indexing the page or directory where it's located.
The document size is too large The maximum page size is 10 MB.
Document format not supported HTML documents and files of the following types are supported: PDF, DOC/DOCX, XLS/XLSX, PPT/PPTX (MS Office); ODS, ODP, ODT, ODG (Open Office); RTF, TXT, and SWF.
Invalid response code Standard HTTP status codes are supported.
The document language isn't recognized. List of supported languages.
The document has no content to index The page is empty or has no content that can be indexed by the robot.
Page content is missing

Check the following in your server settings:

  • HTTP header. For example, if it contains "Content-length: 0", the robot will fail to index the page.
  • The size of the page content. It must be greater than 0 and match the HTTP header.
Sample message