How do I check if a site is accessible to users and indexing bots?

To check if your site pages are accessible to users and indexing bots, you can use the Yandex Webmaster tools.

Site diagnostics

On the Website optimization → Site diagnostics page, you can see if the site has any issues that make it harder to access it.

For example, if the indexing bot can't access your site due to a server error, the site may completely disappear from search results. For the same reason, your site may be unavailable to visitors.

In addition, you can view your diagnostic results in a widget on the Summary page.

Learn more:

• What to do if problems are detected on your site

Site security

Sites infected with viruses are typically inaccessible to visitors because antivirus software blocks their pages in the browser. Hosting providers often disable infected sites making them inaccessible to both users and indexing bots.

To check if there are any security threats on your site, go to Website optimization → Security and violations.

Learn more:

• What should I do if a site security threat is found?

Page check

By navigating to Tools → Check page, you can:

• View how a page is displayed on mobile and desktop devices: a screenshot is available in the Audit results tab. You can also check optimization for mobile devices using the Audit pages for mobile compatibility tool: go to Tools → Audit pages for mobile compatibility.

• Learn which text is available to the indexing bot: go to Page version in the database → Page content.

• Find out which response code (HTTP status code) is returned by the server — by going to Technical information. You can also see the response code as well as the page content accessible to the indexing bot on the Tools → Server response check page.

  If you get the 200 OK response code, this means everything is fine at the moment of checking and the page is accessible to indexing bots and site visitors. Pages with other response codes are inaccessible to bots and users and can't appear in search results.

  For information on what issues correspond to codes different from 200 OK and how to troubleshoot them, see HTTP status codes reference.

Learn more:

• Page status check
• Sites for mobile devices
• Checking the server's response

Checking the robots.txt file

To check if the robots.txt file unintentionally prevents a page from indexing by bots:

1. Go to Tools → Robots.txt analysis.
2. Enter an address in the Check if URLs are allowed field and click Check.

Learn more:

• Using robots.txt