A/B Testing Reliability Calculator
p-value
for additional investments
Segment | Proportion | Expenses | Conversions | Clicks |
|---|---|---|---|---|
A | % | |||
B | % |
CPA | Relative expenses | Relative conversions | Relative clicks | Relative CPA | Relative margin of error for expenditures | Relative margin of error for conversions | Profit | Relative profit | Relative margin of error for profit | Sigma deviation | Probability of this segment outperforming segment A |
|---|---|---|---|---|---|---|---|---|---|---|---|
— | — | — | — | — | — | — | — | — | — | ||
— | — | — | — | — | — | — | — | — | — | — | — |
Is it worth implementing? |
|---|
— |
How to use the calculator
In the table above, you should specify:
→ data of tested segments — the size as a percentage, expenses in any currency, the number of conversions and clicks. In line А, enter the value of the segment that you want to compare to the results of other tests. Fill out the next lines with data from the other segments, their number is unlimited. Empty fields will be filled out automatically.
→ target value level (p-value) — the value reflecting the level of your confidence in the results of an experiment. We recommend setting 80%. Setting a p-value that is too high, e.g. 99%, raises confidence in the results of the experiment, but lowers the number of possible improvements.
→ maximum rigidity coefficient for additional investments — the value reflecting how many times you are ready to increase your advertising investment in order to get more conversions in comparison to segment A. If you are ready to increase your budget for promotion, enter a value higher than 1.
How to read the results
The indicator “Yes” indicates that the results of the given segment are better than those of segment A. The settings for this segment performed better than those in segment A.
The indicator “No” means that the results of the given segment are worse than the results of segment A.
The indicator “?” (“Undetermined”) indicates that there is no statistical difference between A and the segment being compared and we cannot determine which one is best. There can be various reasons for such results — from the segment size to the number of clicks, but it’s almost impossible to say with certainty.
Example
The calculator highlighted the A/B test results in segment B in green.
Conclusion: There is an 86.04% likelihood that the indicators in segment B are better than the data in segment A.
Experiment with the new A/B tests in Yandex.Direct
There is an endless number of scenarios for experimentation — from changing campaign settings to testing different media plans.
Using A/B testing in Yandex.Direct, you can separate your audience into two segments and text different strategies, e.g. click optimization or Average ROI. The calculator tells you which strategy is best for your campaign.