Builds A/B tests

To see how players react to game updates, run an A/B experiment. Through the Developer Console, add a new game build — it will be shown to half of your audience for two weeks. When testing concludes, compare the metrics of the experimental build with the main version’s performance to decide whether to update your app.

Warning

Experiments can only be launched for already published games.

If external factors influence testing, results will be unreliable. To prevent this:

  • Avoid publishing new game versions during the experiment
  • Don’t run other A/B tests simultaneously

For smaller changes, use A/B testing with flags.

Step 1. Preparation

Prepare an experimental build that meets:

Upload it via the Yandex Games Console:

  1. Select your game.
  2. Navigate to Builds A/B Tests tab.
  3. Click Start A/B Test.
  4. Complete Experiment name and What’s new fields. Describe differences from the main version — this speeds up moderation.
  5. Click Choose file to upload the game archive.
  6. Click Submit request. Use Edit submission if needed.
  7. Wait for build verification (≈ 5 minutes), then refresh:
    • If passed: Build passed the check appears.
    • If failed: Click Back to submission, fix issues, and reupload.
  8. Click Submit for moderation.

Step 2. Moderation

Note

You can withdraw a submission within the first 2 hours.

Results appear in Builds A/B Tests within 3–5 days:

The experiment starts automatically — 50% of users see the experimental version.

Moderation rejected the build for violating requirements. You’ll receive an email with:

  • Specific rejection reasons
  • Links to violated requirements
  • Screenshots/recordings if applicable

Before resubmitting:

  1. Review the requirements and testing methodologies.
  2. Implement fixes.
  3. Retest on devices matching moderators’ models (specified in notifications).

Resubmission delays increase by 24 hours after each rejection. For help, contact support.

Step 3. Launch

After launch, the Builds A/B Tests tab shows:

  • Experiment start/end dates
  • Metrics (visible after a few days)

Comparison metrics:

Metric

Description

Timespent per player

Average time in minutes a player spends in the game per day.

Interstitial shows per player

Average number of interstitial ads shown per player per day.

Rewarded shows per player

Average number of rewarded ads shown per player per day.

In-app purchases per player

Average number of in-app purchases per player per day.

Ratio of players with in-app purchase*

Percentage of paying users from the app's daily audience.

Conversion To Play

Percentage of game sessions lasting more than 60 seconds.

Ad revenue delta*

Difference in Yandex Advertising Network revenue between test and control groups, as a percentage of control group revenue.

In-App revenue per player*

In-app purchase revenue per player per day (in rubles).

* Metric visibility is restricted to game owners and developers with View income permissions.

Color coding:

  • Green: Positive statistically significant result
  • Red: Negative statistically significant result
  • No color: Inconclusive impact

Warning

For reliable player activity data, run experiments for at least one week.

Tests auto-complete after 14 days. To end early: Stop → Yes, stop it.

Step 4. Results

Post-experiment options:

  • Publish: Make the new build primary (no re-moderation)
  • Don’t publish: Revert to the original build

Confirm with Close.

Multiple influencing factors make it impossible to isolate the tested changes’ effects.

The build must match the draft’s genre, age rating, tags, and promotional materials.

You may end tests early, but one week minimum is required for statistical significance.

ZIP requirements:

  • ≤ 100 MB
  • Single index.html file only.

95% confidence the improvement relates to build changes.

95% confidence the decline relates to build changes.

Previous
Release updates
Next
Flags