Builds A/B tests

To see how players react to game updates, run an A/B experiment. Through the Developer Console, add a new game build. Specify the audience percentage and experiment duration. When testing concludes, compare the metrics of the experimental build with the main version’s performance to decide whether to update your app.

To study how smaller changes affect game metrics, run an A/B experiment with flags.

Step 1. Preparation

Prepare a new, experimental build. It must comply with the platform requirements and draft materials.

Upload it via the Yandex Games Console:

1. Select your game.

2. Navigate to Builds A/B Tests tab.

3. Click Start A/B Test.

4. Complete Experiment name and What’s new fields. Describe differences from the main version — this speeds up moderation.

5. Select the desired values in the Audience and Duration fields.

6. Click Choose file to upload the game archive.

7. Click Submit request. Use Edit submission if needed.

8. Wait for the build check. Refresh the page in approximately 5 minutes:

   • If passed: Build passed the check appears.
   • If failed: Click Back to submission, fix issues, and reupload.

9. Click Submit for moderation.

   Warning

   Only one moderation can be in progress at a time: draft, creative A/B test, or build A/B test. While the build for an A/B test is being reviewed, other types of moderation will be unavailable.

Step 2. Moderation

Results appear in Builds A/B Tests within 3–5 days:

Step 3. Launch

After the experiment launches, testing data will appear on the Builds A/B Tests tab: the experiment start date and the date when it will automatically end. The first test results will appear in a few days and will be displayed in graphs and tables.

Comparison metrics:

Metric	Description
Timespent per player	Average time in minutes a player spends in the game per day.
Conversion To Play	Percentage of game sessions lasting more than 60 seconds.
Interstitial shows per player	Average number of interstitial ads shown per player per day.
Rewarded shows per player	Average number of rewarded ads shown per player per day.
Ad revenue delta*	Difference in Yandex Advertising Network revenue between test and control groups, as a percentage of control group revenue.
In-app purchases per player	Average number of in-app purchases per player per day.
Ratio of players with in-app purchase*	Percentage of paying users from the app's daily audience.
In-App revenue per player*	In-app purchase revenue per player per day (in rubles).

* Metric visibility is restricted to game owners and developers with View income permissions.

Color coding:

• Green: positive statistically significant result;
• Red: negative statistically significant result.

If metrics are not colored in any way, it is impossible to definitively determine whether the experiment affects the user.

The countdown will begin after moderation review. The experiment will automatically end after the time you selected in the Duration field. To end it early, click Stop → Yes, stop it.

Step 4. Results

After the experiment ends, click:

• Publish to make the new build the main one. You don't need to go through moderation again.
• Don’t publish to end the experiment and stop showing users the experimental build.

After making your decision, click Close.

Since user behavior will be influenced by multiple factors, it will be impossible to determine the role of the originally tested changes.

The new build must match the genre, age rating, tags, and promotional materials specified in the draft.

Available options:

• 10% (5%/5%);
• 20% (10%/10%);
• 50% (25%/25%);
• 100% (50%/50%).

Available options:

• 7 days;
• 14 days;
• 21 days;
• 28 days.

The ZIP archive must:

• be no larger than 100 MB;
• contain only one index.html file.

With 95% probability, the metric improvement is related to changes in the new build.

With 95% probability, the metric decline is related to changes in the new build.