Why should I avoid duplicate pages?

What are duplicate pages?

Duplicate pages are pages with nearly or completely identical text content that belong to the same site but have different URLs.

For example, a home page can have multiple site addresses:

• https://example.com/;
• https://www.example.com/;
• https://example.com/index;
• https://example.com/index.html;
• https://example.com/?utm_source=link&utm_medium=source-example&utm_campaign=partner-offer.

For pages with matching text, the indexing bot creates a group of duplicates. It then selects one page from this group to be displayed in search results. Occasionally, the bot may change its choice to another duplicate.

Why should I avoid duplicate pages?

• The bot indexes multiple pages instead of one. Crawling duplicate pages wastes time as well as the resources of your site and Yandex Search.
• It may take longer to index new pages.
• Duplicate pages may compete with each other in search results.
• The indexing bot can consider a duplicate and exclude from search results a landing page that is important for your site.

Why are there duplicate pages?

Duplicate pages can emerge due to:

• Specific features of your content management system (CMS). For example, page URLs may have or have not a trailing slash (/).
• Web server settings that make site pages accessible over HTTP or HTTPS and with or without the www prefix.
• Adding GET parameters to links, such as tracking UTM tags used by advertising systems.
• The same page appearing in different site sections under different URLs.

In Yandex Webmaster, can I check which pages Yandex Search considers duplicates?

To get a list of duplicates, use the Indexing → Searchable pages tool: open the Excluded pages tab, find the Status column, and apply the Duplicate filter. For more details, click the three dots.

To see if a specific page is a duplicate, insert its address in the URL filter.

To find duplicates emerged due to adding GET parameters to links, run diagnostics: Website optimization → Site diagnostics. Information about duplicates will appear in the critical issues section.

In addition, Yandex Webmaster flags these issues on the Summary page.

Learn more:

• Searchable pages
• Errors

How to remove duplicate pages from Yandex Search?

• Set up redirects: from alternate site addresses to the primary one, and from duplicates to the desired page.
• In the page code, specify which of the duplicate pages you want to include in search results using the rel="canonical" attribute.
• Use the robots.txt file to prevent the duplicates from being indexed.
• Prevent the duplicates from being indexed by adding the noindex rule to the robots meta tag in the page code.

Learn more:

• How to get rid of duplicate pages
• What is a site move, and how do you do it?
• Primary and alternate site addresses

Ungrouping

The owner of a site that has subdomains and often appears at the top of search results may request to reclassify their domain as a web portal through Yandex Webmaster. To do this, you have to provide a description of the services on the subdomains and their owners.

Learn more:

• Ungrouping