Duplicate pages

Pages are considered duplicate if they're available at different URLs, but have the same content. In this case, the indexing robot groups such pages as duplicates. Only one of the pages is listed in the search results — the one the robot will choose.

Note. Duplicates are pages within the same site. For example, pages on regional subdomains with the same content aren't considered duplicates.

Duplicate pages appear for a variety of reasons:

  • Natural. For example, if a page with a product description is available in several categories of an online store).
  • Related to the site features or its CMS.
To find out if your site has duplicates:
  1. Go to the Pages in search page in Yandex.Webmaster and choose All pages in the table.
  2. Download the archive — choose the file format at the bottom of the page. In the file, duplicate pages have the DUPLICATE status. Learn more about statuses
To check if the duplicate page has been removed from the search, check the list of excluded pages.
Note. A duplicate page can be either a regular site page or a fast version of it, such as an AMP page.

To have the right page in the search results, indicate it for the Yandex robot. This can be done in several ways depending on the URL type.

The page URL, including the page path, is different.

Example for a regular site:

http://example.com/page1/ and http://example.com/page2/

In this case:

Example for a site with AMP pages:

http://example.com/page/ and http://example.com/AMP/page/

In this case add the Disallow directive in the robots.txt file to prevent the duplicate page indexing.

The URL displays the URL of the site's main page.
https://example.com and https://example.com/index.php

In this case:

The URL is present or is missing /
http://example.com/page/ and http://example.com/page

In this case, set up a 301 redirect from one duplicate page to another. In this case, the target of the redirect will be included in the search results.

GET parameters differ in the URL.
http://example.com/page/, http://example.com/page?id=1 and http://example.com/page?id=2

In this case:

URL tags (UTM, from, and so on) differ
http://example.com/page?utm_source=link&utm_medium=cpc&utm_campaign=new and http://example.com/page?utm_source=instagram&utm_medium=cpc

In this case, add the Clean-param directive to the robots.txt file so that the robot ignores the parameters in the URL.

The URL contains the AMP page parameters
http://example.com/page/ and http://example.com/page?AMP

In this case, add the Clean-param directive to the robots.txt file so that the robot ignores the parameters in the URL.

The robot learns about changes the next time it visits your site. As soon as that happens, the page that shouldn't be included in the search will be excluded from it within three weeks. If the site has many pages, this may take longer.

You can check that the changes have come into effect in the Pages in search section of Yandex.Webmaster.

If you followed the above recommendations but the changes didn't affect the search results after three weeks, fill out the form below. In the form, specify the sample pages.

Pages with different content can be considered duplicates if they responded to the robot with an error message (for example, in case of a stub page on the site). Check how the pages respond now. If pages return different content, send them for re-indexing — this way they can get back in the search results faster.

To prevent pages from being excluded from the search if the site is temporarily unavailable, configure the 503 HTTP response code.