Builds A/B tests

To see how players react to game updates, run an A/B experiment. Through the Developer Console, add a new game build. Specify the audience percentage and experiment duration. When testing concludes, compare the metrics of the experimental build with the main version’s performance to decide whether to update your app.

Warning

Experiments can only be launched for already published games.

If external factors influence testing, results will be unreliable. To prevent this:

  • Avoid publishing new game versions during the experiment
  • Don’t run other A/B tests simultaneously

For smaller changes, use A/B testing with flags.

Step 1. Preparation

Prepare a new, experimental build. It must comply with the platform requirements and draft materials.

Upload it via the Yandex Games Console:

  1. Select your game.
  2. Navigate to Builds A/B Tests tab.
  3. Click Start A/B Test.
  4. Complete Experiment name and What’s new fields. Describe differences from the main version — this speeds up moderation.
  5. Select the desired values in the Audience and Duration fields.
  6. Click Choose file to upload the game archive.
  7. Click Submit request. Use Edit submission if needed.
  8. Wait for build verification (≈ 5 minutes), then refresh:
    • If passed: Build passed the check appears.
    • If failed: Click Back to submission, fix issues, and reupload.
  9. Click Submit for moderation.

Step 2. Moderation

Note

You can withdraw a submission within the first 2 hours.

Results appear in Builds A/B Tests within 3–5 days:

The experiment started automatically: the selected percentage of users began seeing the new, experimental version of the game, while the rest continued to see the regular version.

Moderation rejected the new build because the changes made to the game violate the requirements. You will receive an email from the moderation team at the address specified in your Yandex ID, which will indicate the specific reasons for rejection with links to the violated requirements. The moderation team will attach files with recordings or screenshots of the violations.

Before resubmitting:

  1. Review the requirements and testing methodologies.
  2. Implement fixes.
  3. Retest on devices matching moderators’ models (specified in notifications).

Resubmission delays increase by 24 hours after each rejection. For help, contact support.

Step 3. Launch

After launching the experiment, the Builds A/B Tests tab will display testing data: the experiment start date and the date when it will automatically end. The first test results will appear in a few days and will be displayed in graphs and tables.

Comparison metrics:

Metric

Description

Timespent per player

Average time in minutes a player spends in the game per day.

Interstitial shows per player

Average number of interstitial ads shown per player per day.

Rewarded shows per player

Average number of rewarded ads shown per player per day.

In-app purchases per player

Average number of in-app purchases per player per day.

Ratio of players with in-app purchase*

Percentage of paying users from the app's daily audience.

Conversion To Play

Percentage of game sessions lasting more than 60 seconds.

Ad revenue delta*

Difference in Yandex Advertising Network revenue between test and control groups, as a percentage of control group revenue.

In-App revenue per player*

In-app purchase revenue per player per day (in rubles).

* Metric visibility is restricted to game owners and developers with View income permissions.

Color coding:

  • Green: positive statistically significant result;
  • Red: negative statistically significant result.

If metrics are not colored in any way, it is impossible to definitively determine whether the experiment affects the user.

Warning

For reliable player activity data, run experiments for at least one week.

The countdown will begin after moderator approval. The experiment will automatically end after the time you selected in the Duration field. To end early: Stop → Yes, stop it.

Step 4. Results

Post-experiment options:

  • Publish: Make the new build primary (no re-moderation).
  • Don’t publish: Revert to the original build.

Confirm with Close.

Multiple influencing factors make it impossible to isolate the tested changes’ effects.

The build must match the draft’s genre, age rating, tags, and promotional materials.

Available options:

  • 10% (5%/5%);
  • 20% (10%/10%);
  • 50% (25%/25%);
  • 100% (50%/50%).

Available options:

  • 7 days;
  • 14 days;
  • 21 days;
  • 28 days.

ZIP requirements:

  • ≤ 100 MB
  • Single index.html file only.

95% confidence the improvement relates to build changes.

95% confidence the decline relates to build changes.

Previous
Release updates
Next
Flags