How do I exclude pages from the search?

Sometimes you need to exclude a site page from search results, for example, if it contains confidential information, is a duplicate of another page, or was deleted from the site.

  1. Step 1. Prohibit the page or directory indexing
  2. Step 2. Speed up the page removal
  3. How do I return a page to the search results?
  4. FAQ

Step 1. Prohibit the page or directory indexing

There are several ways to do this:

If the page is removed from the site
If the page should not be displayed in the search

To check whether the instructions in the robots.txt file are correct, use the Robots.txt analysis tool.

Note. When crawling the site, the robot first accesses the robots.txt file if there is one, and then crawls the site pages. Page-by-page crawling may take some time. If you want to delete many pages from the search, add the Disallow directive for the desired pages.
The robot's behavior when excluding a page with different methods
Exclusion method The robot's behavior
Prohibition in the robots.txt file The robot stops accessing the page within 24 hours.
HTTP status with the 404, 403 or 410 code The robot continues to visit the page for some time to make sure that its status doesn't change. If the page remains unavailable, the robot stops crawling it.
The robots meta tag with the noindex directive
The robot's behavior when excluding a page with different methods
Exclusion method The robot's behavior
Prohibition in the robots.txt file The robot stops accessing the page within 24 hours.
HTTP status with the 404, 403 or 410 code The robot continues to visit the page for some time to make sure that its status doesn't change. If the page remains unavailable, the robot stops crawling it.
The robots meta tag with the noindex directive

When the robot visits the site and finds out that it is prohibited from indexing, the page disappears from the search results within a week. The URL of the deleted page is displayed in the list of excluded pages on the Indexing → Pages in search page in Yandex.Webmaster.

Excluding pages that violate copyright from the search isn't the robot's priority task. To exclude a page from the search, use the methods described in this section.

Pages excluded from search results can be displayed in Yandex.Webmaster until the next site crawl.

Step 2. Speed up the page removal

To speed up the page removal from the search, tell Yandex to remove it, without waiting for the planned robot crawl.

  • If your site isn't added or isn't verified in Yandex.Webmaster:

    1. Go to the Remove pages from search results page in Yandex.Webmaster.
    2. Enter the URL of the page to exclude in the field, for example http://example.com/page.html.
    3. Click the Remove button.

    To exclude multiple pages from the search, remove them one by one.

  • If your site is added to Yandex.Webmaster and you confirmed your site management rights:

    1. Go to the Tools → Remove pages from search results page.
    2. Set the radio button to By URL.
    3. Enter the page URL in the field, for example http://example.com/page.html.
    4. Click the Remove button.

    You can specify up to 500 URLs per site per day.

After the URL is sent to Yandex.Webmaster, you can track status changes on the Tools → Remove pages from search results:

Status Description
“In the delete queue” The robot checks the server response and if the page is prohibited from indexing. The check can take several minutes.
“In progress” The robot checked the page. The page will be removed from search results within 24 hours.
“Deleted” The page was removed from the search results.
“Rejected” The page is allowed for indexing or when the robot accesses the page URL, the server response is different from 404 Not Found, 403 Forbidden or 410 Gone.
Status Description
“In the delete queue” The robot checks the server response and if the page is prohibited from indexing. The check can take several minutes.
“In progress” The robot checked the page. The page will be removed from search results within 24 hours.
“Deleted” The page was removed from the search results.
“Rejected” The page is allowed for indexing or when the robot accesses the page URL, the server response is different from 404 Not Found, 403 Forbidden or 410 Gone.

How do I return a page to the search results?

Remove the indexing prohibition: the Disallow directive in the robots.txt file or the noindex meta tag. The pages return to the search results when the robot crawls the site and finds out about the changes. This may take up to three weeks.

FAQ

I configured redirect to the new pages, but the old ones are still displayed in the search results

If you use a redirect, the robot will gradually track redirects and the old pages will disappear from the search results as it crawls the site. For the robot to learn about the changes faster, send the pages for reindexing.

If the page URLs change because you changed the site's domain name, the search data update may take more than a month. Check if the mirrors are configured correctly.

Tell us what your question is about so we can direct you to the right specialist:

The site or individual pages are prohibited in robots.txt, but are still in the search
The old pages are deleted and redirect to the new ones, but are still present in the search

If you followed the instructions when deleting the site pages and more than two weeks passed after the last crawl by the Yandex robot, fill out the form below:

If you used a redirect, submit pages for reindexing.