How do I exclude pages from the search?

Sometimes you need to exclude a site page from search results, for example, if it contains confidential information, is a duplicate of another page, or was deleted from the site.

  1. Prohibit the page or directory indexing
  2. Speed up the page removal
  3. How do I return a page to the search results?
  4. FAQ

Prohibit the page or directory indexing

There are several ways to do this:

If the page is removed from the site
If the page should not be displayed in the search

To check whether the instructions in the robots.txt file are correct, use the Robots.txt analysis tool.

Note. When crawling the site, the robot first accesses the robots.txt file if there is one, and then crawls the site pages. Page-by-page crawling may take some time. If you want to delete many pages from the search, add the Disallow directive for the desired pages.
The robot's behavior when excluding a page with different methods
Exclusion method The robot's behavior
Prohibition in the robots.txt file The robot stops accessing the page within 24 hours.
HTTP status with the 404, 403 or 410 code The robot continues to visit the page for some time to make sure that its status doesn't change. If the page remains unavailable, the robot stops crawling it.
The robots meta tag with the noindex directive

When the robot visits the site and finds out that it is prohibited from indexing, the page disappears from the search results within a week. The URL of the deleted page is displayed in the list of excluded pages on the Indexing → Pages in search page in Yandex.Webmaster.

Excluding pages that violate copyright from the search isn't the robot's priority task. To exclude a page from the search, use the methods described in this section.

Pages excluded from search results can be displayed in Yandex.Webmaster until the next site crawl.

Speed up the page removal

To speed up the page removal from the search, tell Yandex to remove it, without waiting for the planned robot crawl.

  • If your site isn't added or isn't verified in Yandex.Webmaster:

    1. Go to the Remove pages from search results page in Yandex.Webmaster.
    2. Enter the URL of the page to exclude in the field, for example http://example.com/page.html.
    3. Click the Remove button.

    To exclude multiple pages from the search, remove them one by one.

  • If your site is added to Yandex.Webmaster and you confirmed your site management rights:

    1. Go to the Tools → Remove pages from search results page.
    2. Set the radio button to By URL.
    3. Enter the page URL in the field, for example http://example.com/page.html.
    4. Click the Remove button.

    You can specify up to 500 URLs per site per day.

After the URL is sent to Yandex.Webmaster, you can track status changes on the Tools → Remove pages from search results:

Status Description
“In the delete queue” The robot checks the server response and if the page is prohibited from indexing. The check can take several minutes.
“In progress” The robot checked the page. The page will be removed from search results within 24 hours.
“Removed” The page was removed from the search results.
“Rejected” The page is allowed for indexing or when the robot accesses the page URL, the server response is different from 404 Not Found, 403 Forbidden or 410 Gone.

How do I return a page to the search results?

Remove the indexing prohibition: the Disallow directive in the robots.txt file or the noindex meta tag. The pages return to the search results when the robot crawls the site and finds out about the changes. This may take up to three weeks.

FAQ

I configured redirect to the new pages, but the old ones are still displayed in the search results

If you use a 301 redirect, the robot will gradually track redirects and the old pages will disappear from the search results as it crawls the site. For the robot to learn about the changes faster, send the pages for reindexing.

If you used a temporary redirect (with the HTTP status code 302 or 303), the search may contain pages with the old URLs. You can check the configured redirect using the Server response check tool.

If the page URLs change because you changed the site's domain name, the search data update may take more than a month. Check if the mirrors are configured correctly.