Setup tools

If you need to fine-tune your experiment, take advantage of the additional setup tools.

Sample size calculator

Using the sample size calculator, you can assess whether the experiment's result will be statistically significant with the given audience size.

This tool estimates the MDE to show the smallest change in a metric. You can use it to determine the likelihood of capturing small changes in the experiment with the given data and error levels. The smaller the MDE, the more subtle changes your experiment can detect. To reduce the MDE, increase the sample size or the experiment duration. If the MDE is high, you'll only be able to spot significant changes. You can conduct such an experiment on a small audience.

Fill out the calculator fields to match your app and experiment details:

• Total monthly visits: Traffic to experiment pages based on target conditions, excluding sample sizes. To calculate the value of this field, use data from Yandex Metrica.

• Duration: Duration of the experiment in days. Corresponds to the duration in the Date range field.

• Audience share: Sample size, corresponds to the Audience share field.

• Variants: Number of variants in the experiment, can take values from 2 to 26.

• Metric conversion: Percentage of users converted in the control variant. What action counts as a conversion depends on the exact metric. If you want to get statistically significant results for all metrics, specify the minimum conversion here.

Borders of the detected effect show which metrics will be statistically significant. The values outside these borders will be significant: lower than the red border and greater than the green border. Indicator values in-between can be random deviations from the conversion of the control variant. If this range is too broad and you're interested in less pronounced changes in metrics, try adjusting the conditions of your experiment.

Experiment testing

To make sure that your changes are applied, under Experiment testing, click Test next to each variant. The site version associated with the selected variant opens at a link generated by Varioqub. You can share this link within 24 hours of its generation. Clicking Test again generates a new link.

Editing a running experiment

You can edit a running experiment.

1. In Yandex Metrica, go to Varioqub → Experiments.

2. Click Settings under the experiment name or → View settings next to the name of the experiment you want to edit.

This takes you to the edit page, where you can update any settings configured during creation.

Disabling variants

When editing a running experiment, you can disable specific variants.

Select the variant you want to disable and click Disable next to its name.

Once it's disabled, users originally assigned to this variant will no longer see the experiment changes, as if they were never part of the experiment. Users in other variants won't be affected and will remain in their assigned variant.

Sample experiments

Check out GitHub for examples on how to connect Varioqub and create experiments.

FAQ and troubleshooting

I'm trying to create an experiment, but all the fields are locked.

What if I want to use conditions that are not available in the Varioqub interface?

How do I disable a specific experiment type for an entire site or page?

Contact support

Minimum Detectable Effect.