Excluded pages
Site pages can disappear from the Yandex search results. This can happen for several reasons:
- An error occurred when the robot was downloading or processing the page, and the server response contained a 3XX, 4XX, or 5XX HTTP code. To find the error, use the Server response check tool.
- Page indexing is prohibited in the robots.txt file or using a "noindex" meta tag.
- The page redirects the robot to other pages.
- The page duplicates the contents of another page.
- The page is not canonical.
You can find the list of pages excluded from the search in the Pages in Search section in Yandex.Webmaster.
To have the excluded page displayed in the search again, you should eliminate the reason of its removal. Then use the Page reindexing tool. The pages appear in search results within 1-2 weeks.
Questions and answers about pages excluded from the search
The “Excluded pages” list shows pages that aren't on the site anymore. How do I remove them?
The Excluded pages list in the Pages in Search section shows the pages the robot accessed but didn't index (these may be non-existing pages previously known to the robot).
A page is removed from the excluded list if
- It is not available to the robot for a certain period of time.
- It is not linked to from other pages on the site and external sources.
Excluded pages listed in the service don't affect the site position in the search results.
Pages that are marked as excluded open normally in the browser. What does this mean?
This can happen for several reasons:
- Headers that the robot requests from the server differ from the headers that the browser requests. So excluded pages might open correctly in the browser.
- A page excluded from the search because of a download error disappears from the list of excluded pages only once it is available at the robot's request. Check the server's response for the URL. If the response contains the HTTP 200 OK status, wait for the next robot crawl.