Excluding pages from the search database
Every day Yandex indexes millions of pages and adds them to the search database. To avoid filling it with documents that will never show up in the search results, Yandex analyzes each document using a special algorithm. If a page can't provide one of the most relevant answers to a search, it isn't included in the search database.
Excluding these pages from the search base does not affect the number of visits because the users wouldn't see these pages anyway. In addition, Yandex continues to index and analyze these documents in the same way as all the others. If at some point the algorithm reveals that the page can get into the search results, it will be added to the search engine database.
- Title, description, keywords meta tags on the page are filled in correctly and the page meets all requirements. Why isn't it in the search results?
- Why do pages disappear from the search results and then appear again?
- Why do some pages with a similar topic and content get in the search, while others don't?
Why do pages disappear from the search results and then appear again?
The algorithm checks the pages more than once. It checks the page and calculates the probability of it displaying every time before the search database is updated. The robot indexes excluded pages in the usual way and crawls them regularly. It can detect changes on the requested page and take them into account when calculating the display probability. The algorithm is continually being fine-tuned. This is why pages that were excluded before can appear in the output.
Why do some pages with a similar topic and content get in the search, while others don't?
The algorithm evaluates each page independently from others, even if they belong to the same section. If the page is relevant enough to be displayed in the search results for a specific search in a certain region, it is shown in the search results. The output of the algorithm depends on dozens of indexing factors and hundreds of ranking factors. For example, highly demanded products are displayed in the search, while less suitable products aren't shown in the search output.