Searchable pages

The Yandex search results are regularly updated. Your site pages can appear in the search results and disappear from them.

With the information provided in Indexing → Searchable pages in Yandex Webmaster, you can:

• Monitor the number of pages in the Yandex search results and the ratio of the included and excluded pages.
• Find out the date of the last robot crawl and the search results update.
• Learn about the source from which the page was added to the search results.
• Find out why a page was excluded from the search results.
• Learn the status of new website pages. Even if the pages have been indexed and included in the search immediately after publication, the data in Yandex Webmaster may be updated with a delay. It usually takes a few days for the changes to appear.

By default, the service provides data on the site as a whole. To view the information about a certain section, choose it from the list in the site URL field. Available sections reflect the site structure as known to Yandex (except for the manually added sections).

Page status dynamics

Page information is presented as follows:

• Added and removed — The ratio of the pages included and excluded from the search.
• Excluded — The dynamics of pages excluded from the search.
• History — The dynamics of pages included in the search. Each segment of the graph corresponds to a site section.
• Dictribution — The number of pages included in the search. Each segment of the pie chart corresponds to a site section.

The total number of pages included in the search can exceed the total number of pages in the diagrams (only pages and sections listed on the Site structure page in Yandex Webmaster are counted).

Page changes in the search results

Yandex Webmaster informs you of changes in the search results over the past three months:

• Date of the search results update.
• Whether a page was added to in the search results (the page URL is displayed in green) or removed from them (displayed in blue).
• The reason why the page was removed from the search.

A dramatic change in the number of pages added or participating in search might be related to changes on the site. For example, with changing the site structure or the robots.txt file.

In addition, the service displays general information about the page:

• The date of the last robot crawl.
• The page path from the root directory of the site.
• The page title (the title element).

To view the changes, set the option to Recent changes. Up to 50,000 changes can be displayed.

List of pages included in the search

You can view the list of pages included in the search and the following information about them:

• The date of the last robot crawl.
• The page path from the root directory of the site.
• The page title (the HTML title element).

To view the list of pages, set the option to All pages. The list can contain up to 50,000 pages.

If the list doesn't contain any pages that should be included in the search results, use the Reindex pages tool to inform Yandex about them.

If the list contains pages that shouldn't be included in the search results, see the Deleting a site section.

The source from which the site pages were added to the search results

Yandex’s search engine database receives input about the status of site pages from various sources:

• A site crawl by the indexing bot in normal mode. For example, when it follows known links in the Sitemap file or on other sites.
• Yandex Metrica tag placed on the site;
• IndexNow protocol used on the site.

Information about the data source can be found under Indexing → Searchable pages (Recent changes and All pages).

Updating the source details does not depend on updating the search database or the status of site pages in the search results.

The list of pages excluded from the search

Site pages can disappear from the search results. For more information about why this may happen, see the Why are pages excluded from the search? section.

To view the list of excluded pages (up to 50,000), go to Indexing → Searchable pages and enable Excluded pages. The following information about the pages is displayed:

• The date of the last robot crawl.
• The page path from the root directory of the site.
• Status (the reason why the page was excluded).

A page is no longer listed as excluded from the search some time after the following conditions are fulfilled:

• The page is unavailable for the indexing robot (HTTP server returns 404 Not Found), or there is a 301 redirect set on the page.
• Other resources don't refer to the excluded page.

A page that is removed from the site or doesn't exist can be found in the list of pages excluded from the search, probably because other resources refer to it. To remove the page from the list, disallow indexing in the robots.txt file.

Data filtering

You can quickly find information about a page using filters. To do this, click the icon. You can filter data by all available parameters. For example, you can use URL filtering:

Enter part of a URL

To make a list of pages with a certain text in the URL, choose URL contains from the list and enter the URL fragment.

Use special characters

You can use special characters to match the beginning of the string or a substring, and set more complex conditions using regular expressions. To do it, choose URL matches from the list and enter the condition in the field. You can add multiple conditions by putting each of them on a new line.

For conditions, the following rules are available:

• Match any of the conditions (corresponds to the “OR” operator).
• Match all conditions (corresponds to the “AND” operator).

Characters used for filtering

Character	Description	Example
*	Matches any number of any characters	Display data for all pages that start with https://example.com/tariff/, including the specified page: / tariff / * Using the * character The * character can be useful when searching for URLs that contain two specific elements or more. For example, you can find news or announcements for a certain year: /news/*/2017/.
@	The filtered results contain the specified string (but don't necessarily strictly match it)	Display information for all pages with URLs containing the specified string: @tariff
~	Condition is a regular expression	Display data for pages with URLs that match a regular expression. For example, you can filter all pages with address containing the fragment ~table\|sofa\|bed repeated once or several times.
!	Negative condition	Exclude data for pages with URLs starting with the following string: https://example.com/tariff/: !/tariff/*

The use of characters isn't case sensitive.

The @,!, ~ characters can be used only at the beginning of the string. The following combinations are available:

Operator	Example
!@	Exclude data for pages with URLs containing tariff: !@tariff
!~	Exclude pages with URLs that match the regular expression

Downloading information in a file

You can download the pages information in the XLS or CSV format.

Learn more

• How many site pages are included in search?

A page included in the Yandex search. The may be missing from the search results if it duplicates the content of another page, contains spam or viruses, redirects to another URL, or if there was a loading error on the server side.

A page added to the Yandex search results but then excluded from them.