A/B Testing Reliability Calculator

The calculator allows you to compare the results of A/B tests of several advertising strategies quickly and easily, and determine the most effective set of tools and formats.
The maximum permissible rigidity coefficient
for additional investments
Relative expenses
Relative conversions
Relative clicks
Relative CPA
Relative margin of error for expenditures
Relative margin of error for conversions
Relative profit
Relative margin of error for profit
Sigma deviation
Probability of this segment outperforming segment A
Is it worth implementing?

How to use the calculator

In the table above, you should specify:

data of tested segments — the size as a percentage, expenses in any currency, the number of conversions and clicks. In line А, enter the value of the segment that you want to compare to the results of other tests. Fill out the next lines with data from the other segments, their number is unlimited. Empty fields will be filled out automatically.

target value level (p-value) — the value reflecting the level of your confidence in the results of an experiment. We recommend setting 80%. Setting a p-value that is too high, e.g. 99%, raises confidence in the results of the experiment, but lowers the number of possible improvements.

maximum rigidity coefficient for additional investments — the value reflecting how many times you are ready to increase your advertising investment in order to get more conversions in comparison to segment A. If you are ready to increase your budget for promotion, enter a value higher than 1.

How to read the results

The indicator “Yes” indicates that the results of the given segment are better than those of segment A. The settings for this segment performed better than those in segment A.

The indicator “No” means that the results of the given segment are worse than the results of segment A.

The indicator “?” (“Undetermined”) indicates that there is no statistical difference between A and the segment being compared and we cannot determine which one is best. There can be various reasons for such results — from the segment size to the number of clicks, but it’s almost impossible to say with certainty.


The calculator highlighted the A/B test results in segment B in green.

Conclusion: There is an 86.04% likelihood that the indicators in segment B are better than the data in segment A.

Experiment with the new A/B tests in Yandex.Direct

There is an endless number of scenarios for experimentation — from changing campaign settings to testing different media plans.

Using A/B testing in Yandex.Direct, you can separate your audience into two segments and text different strategies, e.g. click optimization or Average ROI. The calculator tells you which strategy is best for your campaign.

Want to see how this works in practice?
Visit Yandex.Direct