Setup tools

If you need a more detailed configuration when creating an experiment, use the additional setup tools.

Sample size calculator

Using the sample size calculator, you can assess whether the experiment's result will be statistically significant with the given audience size.

This tool calculates the MDE to show the smallest change in the metric. You can use it to determine the likelihood of capturing small changes in the experiment with the given data and error levels. The smaller the MDE, the more subtle changes your experiment can detect. To reduce the MDE, increase the sample size or the experiment duration. If the MDE is high, you'll only be able to spot significant changes. You can conduct such an experiment on a small audience.

Fill in the calculator fields so that they match the information about your app and experiment.

• Total monthly visits: Visits to the experimental pages that met the conditions of the experiment, not accounting for the sample sizes. To calculate the value of this field, use data from Yandex Metrica.

• Duration: Duration of the experiment in days. Corresponds to the duration in the Date range field.

• Audience share: Sample size, corresponds to the Audience share field.

• Variants: Number of variants in the experiment, can be between 2 and 26.

• Metric conversion: Percentage of users converted in the control variant. What action counts as a conversion depends on the selected metric. If you want to get statistically significant results for all metrics, specify the minimum conversion rate here.

Borders of the detected effect show which metric changes will be considered statistically significant. The values outside these borders will be significant: lower than the red border and greater than the green border. Indicator values in-between can be random deviations from the conversion rate of the control variant. If this range is too broad and you're interested in less pronounced changes in metrics, try adjusting the conditions of your experiment.

Testing an experiment

To make sure that your changes are applied, under Experiment testing, click Check next to each variant. The site version associated with the selected variant opens at a link generated by Varioqub. This link can be shared within 24 hours of its generation. Clicking Check again generates a new link.

Editing a running experiment

You can edit a running experiment.

1. In Yandex Metrica, go to Varioqub → Experiments.

2. Click → Edit next to the name of the experiment you want to edit.

This will open a page where you can edit the parameters you set when creating the experiment.

Disabling variants

When editing a running experiment, you can disable its variants.

To do this, select the variant you want to disable and click Disable next to its name.

After disabling, users who were initially included in this variant will no longer see the changes from the experiment, as if they were never part of it at all. Nothing changes for users from other variants, and they'll still remain in their variant.

Examples of experiments

Examples of connecting Varioqub and creating experiments are available on GitНub.

Troubleshooting and questions

I'm trying to create an experiment, but all the fields are locked.

The condition I want to apply isn't available in the Varioqub interface.

How do I disable an experiment type on a page or the entire site?

Contact support

Minimum Detectable Effect.