Why are pages excluded from the search?
Site pages can disappear from the Yandex search results for a number of reasons. To find out why a page was excluded, go to and select Excluded pages. More about the Excluded pages block
| Reason for excluding a page | Solution |
|---|---|
| An error occurred when the robot was loading or processing the page, and the server response contained HTTP status code 3XX, 4XX, or 5XX. | To find the error, use the Server response check tool. If the page is accessible to the robot, make sure that:
|
| Page indexing is prohibited in the robots.txt file or using a meta tag with the noindex directive. | Remove the prohibiting directives. If you didn't place the ban in robots.txt yourself, contact your hosting provider or domain name registrar for details. Also make sure that the domain name isn't blocked due to the registration period expiry. |
| The page redirects the robot to other pages | Make sure that the excluded page should actually redirect users. To do this, use the Server response check tool. |
| The page duplicates the content of another page | If the page is identified as duplicate by mistake, follow the instructions in the Duplicate pages section. |
| The page is not canonical | Make sure that the pages should actually redirect the robot to the URL specified in the rel="canonical" attribute. |
| The site is recognized as a secondary mirror | If the sites are grouped by mistake, follow the recommendations in the Separating site mirrors section. |
| Violations are found on the site | You can check this on the page in Yandex.Webmaster. |
| The page is found to be of poor quality | If the page matches the query well enough, it will be displayed in the search results. If the page doesn't show up in the search results for a long time, it means that it doesn't provide good answers to user queries at the moment. See the recommendations on how to improve the site for users: |
| Reason for excluding a page | Solution |
|---|---|
| An error occurred when the robot was loading or processing the page, and the server response contained HTTP status code 3XX, 4XX, or 5XX. | To find the error, use the Server response check tool. If the page is accessible to the robot, make sure that:
|
| Page indexing is prohibited in the robots.txt file or using a meta tag with the noindex directive. | Remove the prohibiting directives. If you didn't place the ban in robots.txt yourself, contact your hosting provider or domain name registrar for details. Also make sure that the domain name isn't blocked due to the registration period expiry. |
| The page redirects the robot to other pages | Make sure that the excluded page should actually redirect users. To do this, use the Server response check tool. |
| The page duplicates the content of another page | If the page is identified as duplicate by mistake, follow the instructions in the Duplicate pages section. |
| The page is not canonical | Make sure that the pages should actually redirect the robot to the URL specified in the rel="canonical" attribute. |
| The site is recognized as a secondary mirror | If the sites are grouped by mistake, follow the recommendations in the Separating site mirrors section. |
| Violations are found on the site | You can check this on the page in Yandex.Webmaster. |
| The page is found to be of poor quality | If the page matches the query well enough, it will be displayed in the search results. If the page doesn't show up in the search results for a long time, it means that it doesn't provide good answers to user queries at the moment. See the recommendations on how to improve the site for users: |
The robot continues to visit the pages excluded from the search, and a special algorithm checks the probability of displaying them in the search results before each update of the search database. So the page may appear in the search within two weeks after the robot finds out about its change.
If you fixed the reason for excluding the page, send it for reindexing. This way you'll inform the robot about the changes.
Questions and answers about pages excluded from the search
- Headers that the robot requests from the server differ from the headers that the browser requests. So excluded pages might open correctly in the browser.
- It is not available to the robot for a certain period of time.
- It is not linked to from other pages on the site and external sources.
In addition to checking the tags on the page, the algorithm checks if the page content is unique, informative, in-demand and up-to-date, as well as many other factors. However, you should pay attention to meta tags. For example, the Description meta tag and the title element can be created automatically and duplicate each other.
If the site contains a lot of similar products that differ only by color, size or configuration, they may be excluded from the search. The same is true about the pagination pages, product selection and comparison pages, and image pages without text content.
This can happen for several reasons:
The Excluded pages list in the Pages in Search section shows the pages the robot accessed but didn't index (these may be non-existing pages previously known to the robot).
A page is removed from the excluded list if:
Excluded pages listed in the service don't affect the site position in the search results.
