Yandex Blog

Winter Is Coming: Yandex.Weather Nowcasting Helps Users Plan for Precipitation

Like the Seven Kingdoms of Westeros, Russia is known for its vast size and wide range of weather. In particular, many people associate Russia with its distinctive harsh winters that last about five months of the year. The coldest stretch of the Russian winter begins in November and extends through March. During these months, temperatures can regularly reach as low as –34°C (–30°F) in Siberia, one of the coldest regions of Russia that also experiences over 1,000 mm of precipitation in some areas. 

With December fast approaching, weather is shifting and Game of Thrones fans know that “Winter is coming.” To better manage the weather in Russia, over 35 million monthly active users rely on Yandex.Weather, Yandex’s hyperlocal, real-time weather forecasting application, for up-to-the-minute weather updates.

Yandex.Weather incorporates machine learning to provide users with a world best precipitation forecast system. Our weather forecasting technology, Yandex.Meteum, produces 43 percent greater accuracy in predicting precipitation and 25 percent more accurate temperature forecasts than competing weather forecasting services.

While traditional weather forecasting models rely on a combination of data models and surface weather observations to predict weather, Yandex.Weather adds deep learning to the equation. When traditional models are incorrect, for example overestimating rainfall during a cyclone, the temperature is corrected manually with surface observations. Yandex.Weather’s technology, on the other hand, replaces the need for surface observations with the use of deep learning. 

Within the last year, we enhanced our precipitation forecasting technology by incorporating nowcasting.  Yandex nowcasting uses radar imagery to forecast changes in weather using a deep learning neural network to handle transformations, transitions and other changes with the latest radar shot. This technology enables forecasting to be done on a minute-to-minute basis with best-in-class accuracy.   Now users can get push notifications that let them know the moment snow is about to stop in their exact location

Highly accurate, near real-time weather forecasting is clearly useful for people experiencing the variety of weather across Russia. But even within individual cities there is still considerable differences in weather that impacts users’ daily plans. Where traditional weather services generalize results by city, Yandex.Weather shows users hyperlocal results down to the city block.

With hyperlocal forecasting, Yandex.Weather users in one neighborhood may use the application to track a brief sleet storm on the weather map, while users just a few blocks away can receive forecasts showing clear skies in their location. 

Below shows such a scenario that played out yesterday near the Chernyshevskaya metro stop in St. Petersburg, where a user viewing our weather map could have used our service and decided to step inside a coffee shop for some tea during the passing sleet. 

Meanwhile, a user closeby at the Nevskiy Prospect metro using Yandex.Weather might have seen that conditions were fair and recognized they had a half hour to run nearby errands before the sleet reached them. 

As users adjust to shifting weather, their plans impact the demand for nearby businesses including our ridesharing service, Yandex.Taxi. Together, the coming stormfront and surging demand for Yandex.Taxi rides show how much the weather and Yandex nowcasting technology can impact users and businesses.  

With the recent introduction of nowcasting, we can now provide communities with an even more advanced machine learning based forecasting tool to help them more easily navigate their days and stay warm during the “Long Night” of winter!

Celebrating 10 Years of the Yandex School of Data Analysis

Over the years Yandex has launched several education initiatives ranging from learning platforms to high school and master’s level courses. 2017 marks the 10-year anniversary of one of our most impactful educational initiatives, the Yandex School of Data Analysis (YSDA), a free Master's-level program in Computer Science and Data Analysis. In Russia, Yandex is privileged to have access to some of the most talented math and science minds in the world but 10 years ago, Yandex co-founders Arkady Volozh and Ilya Segalovich realized there was a real need to foster these talents and offer students a program for advanced data science.  

 “Together with well-known pattern recognition specialist, Ilya Muchnik, our co-founders considered the need beyond programmers, to programmers with advanced knowledge of the most modern machine learning practices. There was a serious demand for people to lead Yandex and the entire industry down a new path,” says Yandex Director of Human Resources and Director of Computer Science Department at YSDA, Lena Bunina. “And it was clear that we needed to undertake this ourselves.”

Since 2007, the YSDA has been offering Russian students two vigorous years of training from top experts in the most advanced data science topics. Students leave with a profound understanding of theoretical foundations and hands-on experience in applications such as computer vision and  machine translation. “At YSDA, we focus less on theories and more on developing well-rounded students,” says Bunina. “It’s important for students to work in labs, solve problems and receive practical experience.” YSDA students have a unique opportunity to put their data processing and research skills to use as part of YSDA’s collaboration with the LHCb experiment at CERN, the European Organization for Nuclear Research.

YSDA students also have internship opportunities at Yandex that allow them to further expand their training. Interns have the opportunity to work side-by-side with Yandex data scientists helping to deliver our users exceptional customer experiences.

Initially, the school started in Moscow instructing a class of 80 students in the department of Data Analysis. Our academic reputation, coupled with the growing demand for data scientists in today’s AI-centric world,, have created a huge demand for YSDA courses. Last year we received over 4,000 applications from top universities, welcoming 211 students who passed our rigorous entrance exams.

Today, the program has campus branches in Moscow, Yekaterinburg, Novosibirsk and Minsk, plus online offerings in both English and Russian on Coursera and partnerships with some of Russia's leading research institutions and universities.  The Yandex School of Data Analysis offers the Department of Data Analysis and the Department of Computer Science, and with a specialization in Big Data.

At the YSDA, it’s our mission to prepare students to succeed far beyond the walls of our classrooms. “YSDA’s unique blend of science and practice prepares graduates for a wide range of professional options in science, research, product development, analytics and more,” says Misha Levin, Chief Data Scientist at Yandex.Market and YSDA lecturer.

In 2009, we graduated our first 36 students and this past year proudly graduated 123 students.  After ten years, over 600 YSDA graduates are changing the way technology impacts our lives. One of them is Ruslan Mavlyutov, a 2011 YSDA graduate who now works as a Machine Learning Engineer at Apple. “YSDA elevated my machine learning career and I feel privileged to have had the opportunity to study there,” says Mavlyutov. “My peers and I had the unique opportunity to be early ML ‘adopters’ at YSDA enabling us to start and lead ML endeavours at Yandex and other tech giants like Google, Facebook, Microsoft and Apple.” 

“YSDA has created a network of bright minds that pumps ideas across countries, industries and companies. In almost every well-known IT company I can find someone who has studied at the YSDA." Mavlyutov adds, "Those are the people whom you can trust and rely on their expertise.”

We’re excited to be at the forefront educating the next generation of data scientists over the next 10 years. Thank you to all of past and present students and instructors of the Yandex School of Data Analysis! 

Introducing Yandex’s Machine Intelligence and Research Division

Yandex proudly announces the creation of our new Machine Intelligence and Research (MIR) Division. The MIR division will function as a centralized, cross-functional unit to accelerate innovation and unify our core machine learning technologies. The MIR division will also transfer cutting-edge research from our various research teams into Yandex products and services. Yandex has tapped Misha Bilenko to head the new division, which brings together a mix of teams focusing on AI-centered technologies including:

  • MatrixNet and DaNet – Machine learning has always been at the core of Yandex consumer products and information services. In 2009, we launched MatrixNet, our proprietary machine learning platform. Today, MatrixNet is used in nearly every product and service Yandex offers. One important feature of MatrixNet is its resistance to overfitting, which takes into account a very large number of factors when ranking the relevancy of search results. DaNet is the deep neural network (DNN) framework developed at Yandex that provides state-of-the-art runtime performance for many tasks that rely on deep learning.
  • Computer Vision – People learn to recognize objects at a very young age. Machines, on the other hand, must be trained to recognize objects using vast amounts of labeled and unlabeled data. Yandex’s market-leading image recognition technology uses machine learning to detect similar images in visual search results as well as perform a number of high-end vision tasks, from automotive photo analysis for auto.ru, to predicting weather patterns using satellite imagery.
  • Speech – Yandex’s SpeechKit voice recognition technology uses machine learning to help people better communicate with devices and be more productive on the go. SpeechKit technology powers voice commands for Yandex search and is also used in Yandex’s traffic information app, Yandex.Navigator, offering motorists voice activation control. The SpeechKit SDK enables businesses to easily integrate Yandex’s speech technologies in their productivity tools and virtual assistants.
  • Translation – With more than 90 languages in production, Yandex is one of very few companies in the world that has access to enough data to meet today’s high machine translation standards. Yandex.Translate uses machine learning throughout its stack, including unique technology for translating rare languages that don’t have enough written data to use classical methods, instead relying on linguistic structures from related popular languages to fill in the gaps.

From speech-to-speech translation to virtual assistants that chat with people and use cameras to see, the MIR division offers amazing opportunities for synthesis and cross-pollination within Yandex’s machine learning, computer vision, speech and translation technologies. By bringing team members from these core technologies together, the MIR division will improve Yandex’s machine and natural processing capabilities, enhancing its products and services and ultimately delivering consumers and businesses a better experience.

Under Misha Bilenko’s guidance, the unified division will be able to integrate its top research findings across all of Yandex products and services. Misha joins Yandex after 10 years of experience working at Microsoft, where he led the Machine Learning Algorithms team in the Cloud and Enterprise division, following a career in the Machine Learning Group for Microsoft Research. Misha brings a unique blend of leadership skills, research expertise and machine learning knowledge to Yandex. His leadership will be instrumental as the MIR division expands Yandex’s research efforts to experiment with new projects and achieve more long-term goals building the next generation of intelligent products and services.

New Yandex Service Uses Machine Learning for Hyperlocal Weather Forecast

Machine learning is Yandex's core technology. We’ve long been using it in almost all of our services — to answer users’ search queries, for machine translation, ad targeting, personal recommendations, and plotting routes on maps, among others. Since last year, our MatrixNet machine learning algorithm has been utilised for the optimisation of business processes in real enterprises — weopened Yandex Data Factory for this purpose.

Today we announce yet another application of machine learning in a new field for us — weather forecasting. For this we have developed our own forecasting technology Meteum, which will now be used in the web service and mobile application Yandex.Weather available for iOS and Android.

Basic weather forecasts are traditionally constructed using the Navier-Stokes equations. Models for describing weather are extremely complex, as they depend on a multitude of factors. Programs for their calculation consist of hundreds of thousands of lines of code and run on huge supercomputers. Nonetheless, they still make mistakes, so their forecasts need to be fine-tuned. Besides that, the complexity and resource-intensiveness of traditional calculations results in a situation where forecasts are made for relatively large regions and cities. Constructing a precise forecast for, say, a small village would require taking into account a large number of local factors – such as, solar radiation, phase transitions of water vapour, or thermal radiation from the soil. Performing this task using traditional methods is not much less resource-intensive than for a large city, while the number of people using such a forecast is much lower.

Using machine learning allows collating a large volume of historical data about forecasts and actual weather, identifying causality in forecasting errors and correcting them. This is quicker and easier, as it doesn’t require factoring in laws of nature for each new forecast, but simply corrects traditional mathematical models and localises the forecast down to specific latitude and longitude. That’s exactly what Meteum does.

2015-11-24-meteum-640px-72dpi.png

Our new technology uses traditional meteo models to process the initial data, and works with intermediate results using Yandex’s machine learning technology MatrixNet. To calculate the weather, Meteum constantly compares forecast with actual weather conditions — more than 140,000 times a day. To learn about current weather conditions, we use meteorological station data, as well as weather information from other sources indirectly indicating the situation — about 9 terabytes of data every day. One of the sources is our users, who can let us know about discrepancies between forecasts and real weather conditions via the app. The more data we receive from them, the more precise Meteum’s forecasts will become.

Screenshot_2015-11-23-15-12-33.pngfeedback.png


Meteum calculates a new forecast each time a user consults Yandex.Weather on their desktop or mobile device. It locates a person and shows them a fresh forecast for precisely that location. The user can choose another place and time for the forecast to see what the weather will be like around their office in an hour or if it might rain when they go out of town in the evening.

Meteum currently works in 36 regions of Russia, with a possibility to expand to other regions or countries.

Yandex Labs collaborated with Carnegie Mellon University to build a new personalized TV experience

At Yandex Labs, we had a chance to work on a 3-month practicum with students and Professor Ian Lane from Silicon Valley department of Carnegie Mellon University. The project was ambitious but also fun: we wanted to build a new TV experience - personalized and interactive. We developed an application for TV that shows personalized content on a TV screen and allows the users to easily manipulate and interact with the content using hand gestures. The app is still a prototype and is not available for download, but we made this video to share our ideas with you.

The app brings users’ social network streams to their TV screens and allows them to navigate over this information using hand gestures. It is built on Mac OS X platform and we used Microsoft Kinect for gesture recognition.

The application features videos, music, photos and news shared by the user’s friends on social networks in a silent ‘screen saver’ mode. As soon as the user notices something interesting on the TV screen, they can easily play, open or interact with the current media object using hand gestures. For example, they can swipe their hand horizontally to flip through featured content, push a “magnetic button” to play music or video, move hands apart to open a news story for reading and then swipe vertically to scroll through it.

To train gesture recognition, the Carnegie Mellon students together with Professor Ian Lane evaluated several machine learning techniques, which included Neural Networks, Hidden Markov Models and Support Vector Machines (SVM), with SVM showing 20% better accuracy. They put a lot of effort in building a real training set – they collected 1,500 gesture recordings, each gesture sequenced into 90 frames, and manually labeled from 4,500 to 5,600 examples of each gesture. By limiting the number of gestures to be recognized at any given moment and taking into account the current type of content, the students were able to significantly improve the gesture recognition rate.

We have been thinking of controlling a social application with gestures for quite a while. When we found a team of like-minded enthusiasts, we took this opportunity and did a nearly three-month research project. The results of this effort were quite impressive and now we are looking whether we can implement them in a real life application.