Yandex Blog

Forgetting the right to search

The State Duma Committee on Information Policy and Communications has discussed a bill that requires search engine operators to delete hyperlinks to illegal or unreliable information, or even reliable information that refers to events that happened three years ago or more, from their search results on requests from individuals and without a court order.

Internet search is our core business. In more than 15 years in this market, we have put colossal human and financial investments into our search engine, first and foremost, to offer our users search results that are complete, unbiased and useful. If this bill is passed in its current form, a search engine based on these principles will be difficult or even impossible to develop. That is why we feel it is important for us to offer commentary on this bill.

According to its authors, this bill enables any individual to control distribution of unreliable or outdated personal information on the internet. In principle, this gives people a right, which is based on one of the most basic human rights – the right to privacy, including the right to control access to information about oneself. Unfortunately, the procedure offered in this bill does not stop information from being distributed online, but contradicts the basic principles of law and current legislation.

The current law does not permit limiting a person's right to access reliable information. The Constitution of the Russian Federation guarantees everyone the right to freely seek, obtain, transfer, produce and disseminate information by any lawful means (Article 29). The Federal Act ‘On Information, Information Technologies, and Information Protection’ also stipulates an individual’s or organization’s right to search and obtain any information in any form from any source (Article 8). This is exactly what a search engine does – searches for information available through any public source. This bill ignores the right to search for information.

The limitations introduced by this bill reflect imbalance between private and public interests. The need to seek and obtain information often falls within public interest and concerns public figures, whose actions can have an impact on the general public or private lives. This bill impedes people's access to important and reliable information, or makes it impossible to obtain such information. If this bill is passed, the information about a clinic or a doctor, a school or a teacher one is considering to choose, may be impossible to find.

In addition, the procedure for requesting a search engine to remove hyperlinks introduced in this bill opens the door to numerous opportunities for misuse, as it doesn't require any evidence or justification. A search engine, on the other hand, is required to delete an undefined number or hyperlinks to indeterminate web pages. This loophole can very conveniently be used by unscrupulous businesses to undermine their rivals, or by criminals to facilitate fraud.

But even if we assume that it is possible to equal adequate information with inadequate or illegal information in the right to be searched for, one question remains: who will study the information which is searched for, and decide whether it is legal, adequate, relevant or reliable? The bill assigns this role to search engines, while the functions of the court or law enforcement agencies are given to individual commercial organizations. Failure to comply with this role is punished with penalties and litigations.

This bill also ignores the basic principles of information technology and information search. It gives any person the right to request a search engine operator to stop providing hyperlinks to web pages that contain specific information, but it does not require this person to say which hyperlinks should be removed. All they have to do is provide the information, hyperlinks to which they want to be removed. Instead of deleting hyperlinks to specific web pages from search results, a search engine is expected to stop retrieving a piece of information on any search terms and regardless of its location on the internet. For this to become plausible, a search engine operator would have to find all pages containing this information that might appear in any place in search results triggered by any search term that a human mind can come up with. This step alone would take eternity. The next steps would require a search engine operator to make sure that these pages do contain the information hyperlinks to which were requested to be removed, and then confirm that this information is indeed inadequate or older than three years old. It is obvious that this is an impossible task.

Even though the list of flaws of this bill can go further, it doesn't make sense discussing them all at a point when the stipulated procedure itself contradicts the law and is technically impossible.

The current bill is much less well thought through than the Google Spain v ARPD, González (C-131/12) decision by the Court of Justice of the European Union, which has been widely criticized, and which the Russian bill has often been likened to.

The links to be removed from search results mandated by the ruling of the Court of Justice of the European Union are specific, lead to specific information and appear on a narrow class of search terms. Hyperlink erasure is also considered on a case-by-case basis to make sure it does not limit access to important information or alter the balance between private and public interests.

Yandex Takes Leap With Its New Interactive Search Platform - First In Turkey

Yandex has always been there to give answers. But then there comes a moment when giving answers becomes ‘not enough’. A large portion of the five billion searches that come from almost 100 million Yandex users each month are looking for a solution rather than an answer – people want to pay a bill, check-in for a flight, book a show or a visit to the dentist.

Using all our knowledge and expertise, all these years we have been developing products and technologies to figure out what it is exactly that our visitors are looking for and offer this to them in our search results. But even the most relevant of search results doesn’t always give a solution. Even if you can see in search results a link to the website where you can potentially book your movie ticket, you still need to click through to this site and possibly even browse a few pages before you find the booking form.

We have been thinking about offering web users a shortcut precisely to what they need, and we came up with a new concept of a goal-oriented search platform. 

Yandex's new search platform embodies our new approach to search and website engagement, and features an attention-based search results page, where each search result is a standalone block of interactive information – an Island. These blocks are the first step to the user’s search goal and can be anything from factual information to purchase buttons or order forms.


While web users can instantly see and choose the best and most relevant solution to their problem, Islands enable website owners to directly connect with their visitors and enjoy handpicked target audience. Any website would like to see an interested visitor and about 15-20% websites on the Russian internet are using interaction to improve the quality of website experience for their visitors. Our new search platform embraces website owners catering for the interests of internet users. We are offering them an opportunity to choose what information from their site should be featured on Yandex’s search results page. Through our Yandex.Webmaster service, using a free and easy toolset, website owners can markup their pages and see what their Island will look like on Yandex’ search results page right now. Or see whether they need an Island of their own at all.

Right now, Islands are triggered by the same search algorithms as regular snippets, but we are working on giving websites an opportunity to tune their interactive snippets to specific search parameters, such as search terms, user’s search history, their location, and even their device. Prospectively, Yandex users looking for ‘The Great Gatsby’ on their personal computer will see a ‘film review’ Island, those searching for the same on their mobile device will find an Island with the show times at the closest cinema and a booking form, while an Internet-connected TV viewer will receive an Island with a pay-per-view page to purchase this film. This is the next step in Yandex Islands’ future development.

Yandex’s ranking is not affected by Islands, as such. A website’s natural position, however, does depend on the quality of the user experience it provides. The better is the website, the more popular it is with the users, the higher is the ranking. It is quite possible that web users may like a website more because it has an interactive snippet, but the mere fact of having such a snippet won’t propel a website to the top of the search results page. 

By introducing the goal-oriented search, Yandex facilitates interaction between web users and websites. Our mission is to offer solutions to people’s problems by openly sourcing the best of the web and delivering it to web users. We would like to attract the actual content generators and publishers and see them actively participate in this process.

We are rolling out our new experimental search platform, Islands, first, for the eight million of Yandex users in Turkey. Yandex Islands is now publicly available at yandex.com.tr on personal computers and tablets in this country. The mobile version is expected to see the light later this summer. This is the first time we are launching a new search platform on a relatively new market, without first trying it out at home.

The Turkish market is not only new, it also has a young and open-minded demographic, which, we believe, is hard to scare by innovation. The new search platform’s success is largely in the hands of website owners and web publishers. We will focus on educating them about the benefits and possibilities of Yandex Islands in a series of free workshops, which will be open for all and where any website owner can learn how to make an Island of their own. Detailed instructions for creating interactive Yandex snippets for specific website categories, such as hotels, restaurants, airlines, or car services, will soon appear on the Yandex Islands specification pages

Our users in Russia, Ukraine, Belarus and Kazakhstan will see their own Yandex Islands some time later this year.

It May Get Really Personal – we have rolled out our second-generation personalised search program

We all know what it takes to understand another person. It’s a lot. Even if all you need to understand is what a person is looking for online. We have been trying to do this for years. A person’s interests and preferences give a good clue as to what they want to find. We used to look into a user’s search history as far as a few months back to choose for them the search results that would be most relevant to their scope of interests.

Now, we have added to our search algorithm a search history of a few seconds – searches within the current search session. We can now deliver results and search suggestions based on the ‘full picture’ of the user’s search behaviour.


Updating our knowledge of users’ interests once a day allows us to understand their more-or-less stable interests, such as a love of books or football, or that they speak Russian and live in Saint Petersburg. More than half of all searches on Yandex, however, are about something that interests the searcher at the very moment of searching and stops interest them the moment after. To be able to cater to such momentary searches, we now analyse search sessions in real time.

Search queries begin to influence search results within seconds. The search engine can figure out whether a person is looking for a book or a film even before they have finished typing [The Great Gatsby] in the search box.

 

 

 

With the new personalised search program, we can offer relevant results even to those web users who don’t have any search history on Yandex. To instantaneously react to changes in users’ search behavior, we created a real-time data processing system, which processes more than 10 terabytes of data a day, continuously correcting its knowledge of users’ needs.

To make search results as personal as they can be, we first learnt to take into account users’ language preferences and permanent interests. Now, we have learnt how to tell current search intents of our users from their search footprint and give them what they are looking for. Personally.

Search results based on current search interests are available on all Yandex domains globally for all 93 millions of our users. Personalisation that takes into account long-term or medium-term interests works best for searches in Russian, but, just like all previous editions of Yandex’s Personalised Search, the latest version will learn to fetch personlised search results in other languages as well as it receives more search queries in these languages. 

Read more about our Personalised Search program in Products and Technologies.

Yandex has rolled out personalisation for its search results and search suggestions

The search engine now fetches results and delivers search suggestions individually for each of its users based on their interests and preferences.Web users in Russia, for instance, typing on Yandex the query [nevermind] might just as likely be looking for Nirvana's album, as wishing to find out what this word means. Personalised Search would know the difference and would act accordingly.

Yandex’s Personalised Search is based on the user's language preferences, their search history and their clicks in search results. The user's search history tells the search engine what may be currently relevant for this particular user. Someone searching on the internet for free software, books or music is very likely to be interested in this type of content as such. Those users, who frequently visit websites in English, may very well appreciate search results in this very language. As personal preferences tend to change over time, Yandex considers only the relatively fresh search history spanning the period of a few months to offer the users personalised search results and make personalised search suggestions.  

Search suggestions

In contrast to regular search suggestions, personalised search suggestions are targeted individually at every single web user. When guessing what a searcher might want to find, Yandex suggests potential search queries, based on what other people with similar online preferences looked for on the internet. The search engine classes everyone into one of about 400,000 user groups with more-or-less common interests. This classification is fluid – it changes for every user according to the changes in their online behaviour.

In practice, web users repeat about 25% of their search queries and often click the same search results. This behaviour can be interpreted as going to frequently visited websites or viewing popular or personally relevant web documents. Yandex offers users a shortcut to favourite content by showing them recently made queries and their favourite websites in the search suggestions when they type the first letter of their new search. 

When choosing search suggestions for a specific user, Yandex also looks at what searches have been previously made during a whole search session. So, the search engine would know that Christopher Lloyd would probably be a better search suggestion for 'c' in the search box than any other if the searcher has looked for Back to the Future before.

Personal results

Other than making personalised search suggestions, Yandex helps its users to achieve their search goals by providing them with the most relevant search results. In doing that, the search engine uses a special personalised ranking algorithm, which it recalculates on a daily basis according to the ever-changing interests and language preferences of each user.

Personalised ranking algorithm allows the search engine to understand how well each of the fetched results matches the user's expectations. Search results are evaluated and ranked according to their usefulness for a particular web user. The same search query made by two different people would trigger the same results ranked differently to match their individual interests. An inveterate gamer and an art film enthusiast, for instance, will see on top of search results links to websites that are relevant to their respective interests even if they both look for 'Stalker'.


  

The personalised search algorithm works only for those queries that may be related to the user's search history. Yandex won't be able to use a personalised search formula based on the user's preferences if they are looking for something they have never searched before. This formula, however, does work, on average, for about 75-80% search queries. Which queries trigger personalised search depends on the user and on their personal search history.

Personalised search is enabled by default for every more-or-less frequent search user. The more queries the user makes, the better results and suggestions the search engine can provide. Personalised search deactivates if there aren't enough searches on which personalisation can be based, and activates again when queries start coming. Personalisation can also be enabled or disabled manually in the search engine's settings.

Currently, personalisation on Yandex works best for searches in Russian, but as the search engine accumulates search statistics in other languages other users will also be able to feel its effect.