How do I prohibit indexing of the site or pages?
Sometimes it is needed that the site pages or links placed on them don't appear in the search results. You can use the robots.txt file, HTML markup or authorization on the site to hide the site content from indexing,
Prohibiting a site, section or a page from indexing
If some pages or sections of the site shouldn't be indexed (for example, the ones that contain technical or confidential information), use the following methods to restrict access to them:
- In the robots.txt file, specify the Disallow directive.
- Specify the robots meta tag with the noindex or none directive in the HTML code of the site pages. For more information, see Robots meta tag and HTTP header X-Robots-Tag section.
- Use authorization on the site. We recommend this method to hide the home page from indexing. If the main page is prohibited in the robots.txt file or with the noindex meta tag but there are links leading to it, the page can be included in the search results.
Prohibiting page indexing
- Hide part of the page text from indexing
Add the noindex element in the HTML code of the page. For example:
<noindex>text that shouldn't be indexed</noindex>
The element is not sensitive to nesting — it can be located anywhere in the HTML code of the page. You can use the tag in the following format if necessary for making the site code valid:
<!--noindex-->text that shouldn't be indexed<!--/noindex-->
- Hide a link on a page from indexing
In the HTML code of the page, add the rel= " nofollow" attribute in the a element. For example:
<a href="url" rel="nofollow">link text</a>
The attribute works the same way as the meta tag with the nofollow directive in the robots meta tag, but it only applies to the specified link.