Experiment results report

To view a report:

1. In Yandex Metrica, go to Varioqub → Experiments.

2. Under the name of the experiment, click View result.

Ways to use this report

With this report, you can see if the experimental variant has any statistically significant changes in the primary metrics compared to the control variant.

The experiment tests hypotheses:

• H₀: The metric value for this variant doesn't differ from the control one.

• H₁: The metric value for this variant differs from the control one.

The reports use the Mann-Whitney U test with the application of bucketing methods as the test statistic. If there is too little data, discreteness corrections are used.

When the test statistic of a hypothesis takes on the value of H₀, the corresponding row isn't highlighted in the Yandex Metrica report.

Keep in mind that if your hypothesis has a value of H₀, that doesn't mean that the metric doesn't change. The only conclusion that can be made with sufficient certainty is that the effect is smaller than the MDE (minimum detectable effect). You can detect smaller changes in metrics by increasing the audience size and duration of the experiment.

When the test statistic of a hypothesis takes on the value of H₁ with P-value ≤ 0.05, the corresponding row in the Yandex Metrica report is highlighted in color. The highlight color has three possible levels of intensity determined by the P-value. The threshold values for the different intensity levels are 0.05, 0.01, and 0.001.

Report structure and settings

The report data is grouped by the variants included in the experiment. The control variant is placed in the first row.

When creating a new experiment, you can select its primary and secondary metrics. During the course of the experiment, these metrics become available in the report:

Primary metrics

• Time on site: Information about how much time users spent on the site, taking into account the number of sessions.

• Page depth: Number of site page views per session.

• Visual bounce rate. Only the sessions of users selected for the experiment count toward the metrics. For example, if a user clicks a button and immediately leaves the page, the experiment doesn't trigger: such a session is considered a bounce, but not a visual bounce. Therefore, there tends to be fewer visual bounces than Yandex Metrica bounces.

• Audience (count of unique users): Analysis of visitor characteristics. A unique visitor is a user who's assigned a unique site visitor ID (i) for the first time.

• Income of all goals per visit:

  • Revenue for a specific goal.

  • Revenue for all goals.

  • Revenue for a specific step of a multi-step goal.

• Conversion to goal: Percentage of session that complete the selected goal out of the total number of sessions recorded for the variant.

• Conversion to multiple goals: Percentage of sessions with the achievement of the specified number of selected goals out of the total number of sessions in the option.

• Conversion per user to goal (Bayes): Ratio of unique users who achieved the goal to the total number of unique users. This calculation uses the Bayesian method.

• Conversion to step of multi-step goal: Set a sequence of steps that the user takes toward a conversion. These steps can be pageviews or JavaScript events.

• Goal income per visit: When a session goal is completed, Yandex Metrica can record revenue for that goal. This helps you measure cost-efficiency for the goal.

• Step income of multi-step goal per visit: When a step of a multi-step goal per session is completed, Yandex Metrica can count revenue for that goal. This helps you measure cost-efficiency for the goal.

E-commerce metrics

To see these metrics in the report, enable e-commerce.

• Share of buying sessions.
• Average GMV per buying session.
• Average GMV per session.

Advertising metrics

To see advertising metrics in the report, connect a Yandex Metrica tag to your ad. To learn more, see Ad experiments.

The revenue data in Yandex Metrica may differ from the data shown in the advertising dashboard and the actual amounts, as Yandex Metrica and other services have different revenue calculation algorithms.

• Average revenue per visit (YAN): Revenue generated by ad impressions per session.

• Average revenue per user (YAN): Revenue generated by ad impressions per user.

In addition, the report includes:

• Auxiliary metrics. For example, for Conversion for goal, the report shows the numerator and the denominator: the number of sessions that complete the goal and the total number of sessions.

• The value of the metric itself. For example, Conversion for goal is the percentage of converted sessions from the total number of sessions.

• Delta: Difference between the metric values in the experimental and control version.

• Delta (%): Delta expressed as a percentage from the control version's metric value.

• Confidence interval (±2σ): Visualization with the confidence intervals of the experimental and control variants on a number line.

• P-value: Main quantifier of the test statistic result. It represents the probability of getting as extreme or more extreme results under the assumption that the metric value doesn't change (hypothesis H₀ in the example). To learn more, see this article.

  To accept a hypothesis, the P-value is compared with the significance level: P-value <= alpha. The default threshold is alpha = 0.05.

  Keep in mind that alpha sets the probability of type I errors (false positives). It's impractical to set a very low alpha value, as it increases the probability of type II errors (false negatives) and raises the MDE.

• MDE (%): Minimum detectable effect. This is the smallest change in the metric that can be detected with the existing amount of data and with the probability of type I errors alpha = 0.05 and type II errors beta = 0.2. The MDE is expressed as a percentage of a metric value in the control variant. You can lower the MDE by increasing the audience size and duration of the experiment.

• Filters (Add filter):

  Basic plan

  Standard plan

  Extended plan

  Filter	Description
  Without robots	Excludes traffic from web crawler sessions from the report.
  Without bounce visits	Excludes traffic from sessions identified as bounces from the report.
  Only full days	Exclude days when the experiment didn't run for a full day, such as the start or end day of the experiment.

  Filter	Description
  Without robots	Excludes traffic from web crawler sessions from the report.
  Without bounce visits	Excludes traffic from sessions identified as bounces from the report.
  Only full days	Exclude days when the experiment didn't run for a full day, such as the start or end day of the experiment.
  Platforms	Segments data by device. Platforms available for filtering: desktops, mobile phones, tablets, TVs.
  Traffic sources	Use this filter to track the origin of traffic to your site. To do this, select the source type, for example: Direct traffics, Search engine traffic, Click-throughs by site link. The report uses automatic attribution. For more information, see Yandex Metrica Help.
  Operating systems	Segments data by operating system. Operating systems available for filtering: iOS, Windows, GNU/Linux, macOS, Android.
  Gender	Filtering by gender: male or female.
  Age	Segments the audience by age groups: under 18, 18–24, 25–34, 35–44, 45–54, 55 and older.
  Interests	Segments the audience by interests: entertainment and recreation; finance; construction; improvement and repair; health and beauty; real estate; food and drinks; telecom; family and children; business.

  Filter	Description
  Without robots	Excludes traffic from web crawler sessions from the report.
  Without bounce visits	Excludes traffic from sessions identified as bounces from the report.
  Only full days	Exclude days when the experiment didn't run for a full day, such as the start or end day of the experiment.
  Conversion	Filters report data by one of the completed conversions: button click or click to cart.
  Platforms	Segments data by device. Platforms available for filtering: desktops, mobile phones, tablets, TVs.
  Traffic sources	Use this filter to track the origin of traffic to your site. To do this, select the source type, for example: Direct traffics, Search engine traffic, Click-throughs by site link. The report uses automatic attribution. For more information, see Yandex Metrica Help.
  Operating systems	Segments data by operating system. Operating systems available for filtering: iOS, Windows, GNU/Linux, macOS, Android.
  Gender	Filtering by gender: male or female.
  Age	Segments the audience by age groups: under 18, 18–24, 25–34, 35–44, 45–54, 55 and older.
  Interests	Segments the audience by interests: entertainment and recreation; finance; construction; improvement and repair; health and beauty; real estate; food and drinks; telecom; family and children; business.

This report doesn't support Yandex Metrica default settings.

Other metrics

You can expand reports on ongoing or stopped experiments with additional metrics.

This is useful if you selected an irrelevant metric when creating the experiment or if you need to analyze the experiment's impact on other metrics.

1. In Yandex Metrica, go to Varioqub → Experiments.

2. Under the name of the experiment, click View result.

3. In the menu that appears, click the selected metric and choose Other metric.

A list of available metrics opens, excluding those you've already selected. Select the desired metric to see how it was affected by the experiment.

Complete experiment report

You can view and download the complete report on a stopped experiment.

1. In Yandex Metrica, go to Varioqub → Experiments.

2. Click → View full report next to the name of the experiment for which you want to get the complete report.

A page will open with a report that contains complete information about all experiment settings, detailed data for the main metric with a chart, and summarized data for all auxiliary metrics.

• Click Update report to update the report data. Reports can't be updated more than once per hour.

• Click next to the Update report button to print the report.

Publishing a variant of the experiment via personalization

To apply a successful experiment variant, create a personalization from it.

1. In Yandex Metrica, go to Varioqub → Experiments.

2. Under the name of the experiment, click View result.

3. Under the variant with changes, click Create personalization. The personalization draft with the conditions of this variant will open.

4. Click Save.

The changes from the selected variant will be applied to the entire website audience, taking into account the conditions of the experiment. For example, the personalization will include platform or region-specific conditions and URL filters.

To cancel the changes, stop the personalization:

1. In Yandex Metrica, go to Varioqub → Personalizations.

2. Next to the personalization, click → Stop.

FAQ and troubleshooting

The experiment data doesn't match the Yandex Metrica report data.

In my redirect experiment, the test and control values differ significantly.

After deleting cookies from my device, I was in multiple experiments, but the report shows only one session.

My monetization statistics in Yandex Metrica and the partner interface are different.

Contact support

Site user's ID stored in first-party cookies. Make sure to set it in the _ymab_param cookie upon receiving the response. It may be empty during the user's first visit to the site.

We recommend passing it in each API request. The field may be empty in the first request.