How do my site pages appear to indexing bots and users, and how can I influence that?

To ensure that visitors and indexing bots can perceive your site's content easily, use HTML layout features and enhance your texts with images, videos, and other visual elements. To make your site pages easily recognizable among other browser tabs, add a favicon to your site.

To make the site content clearer to indexing bots, add the title and description meta tags to your pages and use semantic markup. If your site has pages that use JavaScript, check how the bot indexes them and make sure that all page content important to you is included in search.

Organizing content

Ensure that your content is organized in a logical manner so that both site visitors and indexing bots can understand it easily.

To organize your page content:

  • Arrange the content so that the main semantic section appears on the first screen where users can see it without scrolling the page.
  • Use a hierarchy of header tags (H1, H2, H3...) to improve navigation and highlight key points in your text.
  • Break the text into paragraphs meaningfully and use lists, tables, and CSS styles. This makes your text visually appealing and easier to understand.
  • Use images and videos. To help bots understand what is shown in images and videos, add descriptions using the alt and title attributes.
  • To ensure faster page load times, avoid bloating your site with unnecessary code and large media files.
  • Make sure that the page's HTML code doesn't contain any serious errors and the page is rendered correctly across various browsers and devices.

Learn more:

"Title" and "description" meta tags

The title meta tag defines the title of a site page. It is displayed in browser tabs and summarizes the page's content for visitors and indexing bots. The description meta tag provides a general description of the page's content for indexing bots.

To use meta tags, add them to your site's code. In most CMS platforms, you can do this using a visual editor.

These meta tags can also be used to generate page snippets shown in search results.

Learn more:

Favicon

A favicon is an icon that appears in browser tabs and helps your site stand out among other tabs. The favicon is also displayed next to your site in search results.

Use a favicon that effectively reflects your site's content, makes your site recognizable, and sets it apart from other sites.

Learn more:

JavaScript page rendering

Sites using JavaScript attract users with their dynamic content. However, due to longer load times and the presence of alternative scenarios, bots might not be able to fully index such sites.

In Yandex Webmaster, you can check how JavaScript pages were indexed and whether all their important content is included in search:

  1. In Yandex Webmaster, go to Indexing → Check page.
  2. Specify the page URL and version (mobile or desktop).
  3. Run the check and wait for the results.
  4. Click Learn more and go to the tab showing the page version in the database.

Yandex Webmaster also features the JavaScript page rendering tool (Indexing → JavaScript page rendering). Use it to:

  • Prohibit indexing bots from executing JavaScript when crawling your site. This might be helpful if you need to reduce the load on your server.
  • See how your site pages are displayed in the browser.
  • Find out which content can be accessed with and without JavaScript execution.

Learn more:

How to hide a page or its part from indexing bots

To hide a site page from indexing bots while keeping it accessible to users:

  • Use the Disallow directive in the robots.txt file.
  • Use the robots meta tag with the noindex or none directive.
  • Require authorization on your site.

If you only need to hide part of the page content, use the noindex or noscript tag.

To see if the page is indexed as you need it, use the Check page tool (Indexing → Check page).

In Yandex Webmaster, you can also check whether the page URL is allowed for indexing or blocked from it: Tools → Robots.txt analysis → Check if URLs are allowed.

Learn more:




You can also go to

Previous
How to monitor and manage indexing?
Next
How do I prevent from indexing the pages that aren't needed in search results?