--- metadata: - name: generator content: Diplodoc Platform v5.55.3 alternate: - https://yandex.com/support/varioqub-app/en/web/stat.md - https://yandex.com/support/varioqub-app/ru/web/stat.md - href: https://yandex.com/support/varioqub-app/en/web/stat.md type: text/markdown title: Markdown version - href: https://yandex.com/support/varioqub-app/en/llms.txt type: text/markdown title: llms.txt --- > **Documentation Index:** Fetch the complete configuration index at https://yandex.com/support/varioqub-app/en/llms.txt # Experiment results report To view a report: 1. Go to AppMetrica. 2. Go to **Varioqub → A/B experiments** and click **View result** under the name of your experiment. ## Ways to use this report {#case} With this report, you can see if the experimental variant has any statistically significant changes in the primary metrics compared to the control variant. The experiment tests hypotheses: - **H0**: The metric value for this variant doesn't differ from the control one. - **H1**: The metric value for this variant differs from the control one. The reports use the _Mann-Whitney U test_ with the application of bucketing methods as the test statistic. If there is too little data, discreteness corrections are used. When the test statistic of a hypothesis takes on the value of **H0**, the corresponding row isn't highlighted in the AppMetrica report. Keep in mind that if your hypothesis has a value of **H0**, that doesn't mean that the metric doesn't change. The only conclusion that can be made with sufficient certainty is that the effect is smaller than the **MDE** (minimum detectable effect). You can detect smaller changes in metrics by increasing the audience size and duration of the experiment. When the test statistic of a hypothesis takes on the value of **H1** with **P-value** ≤ 0.05, the corresponding row in the AppMetrica report is highlighted in color. The highlight color has three possible levels of intensity determined by the **P-value**. The threshold values for the different intensity levels are 0.05, 0.01, and 0.001. ## Report structure and settings {#data} The report data is grouped by the variants included in the experiment. The control variant is placed in the first row. ### Metrics {#metrics} When [creating a new experiment](https://yandex.com/support/varioqub-app/en/web/create.md), you can select its primary and secondary metrics. During the course of the experiment, these metrics become available in the report: **Primary metrics** - **Number of users**: Number of app users for the specified period. - **Number of sessions**: Number of sessions divided by the number of users with at least one session for the specified period. - **Average session duration**: Total duration of all sessions divided by their number (sessions with an undefined duration aren't taken into account). - **Timespent per user**: Time a single user spends in the app. - **Conversion to event "A"(c)**: Ratio of the number of users with event _A_ containing selected parameters _c_ to all users. - **Conversion from event "A"(c) to event "B"(d)**: Number of users with event _B_ containing selected parameters _d_ divided by the number of users with event _A_ containing selected parameters _c_. - **Step-by-step conversion from event "A"(c) to event "B"(d)**: Ratio of users with event _B_ containing selected parameters _d_, which is preceded by event _A_, to users with event _A_ and parameters _c_. {% note info %} In the **Conversion to event "A"(c)**, **Conversion from event "A"(c) to event "B"(d)**, and **Step-by-step conversion from event "A"(c) to event "B"(d)** reports, parameters (_c_, _d_) are optional and may be omitted. If you [add multiple parameters](https://yandex.com/support/varioqub-app/en/web/create.md#metrics) to an event, they are shown next to the event name in the report, for example: **Conversion to event "A"(b→c→d)**. In this example, _A_ is the event, _b_ is the first-level parameter, _c_ is the second-level parameter, and _d_ is the third-level parameter. {% endnote %} **E-commerce metrics** - **In-App ARPU**: Average revenue from purchases made within the app, calculated per user for the specified period. - **E-commerce purchase conversion**: Share of users who made at least one e-commerce purchase out of all people involved in the experiment. - **E-commerce purchase revenue per user**: Average revenue from e-commerce purchases per user for the specified period. **Advertising metrics** - **Ad revenue per user**: Revenue generated by in-app ad impressions per user. In addition, the report includes: - **Delta**: Difference between the metric values in the experimental and control version. - **Delta (%)**: Delta expressed as a percentage from the control version's metric value. - **Confidence interval (±2σ)**: Visualization with the confidence intervals of the experimental and control variants on a number line. - **P-value**: Main quantifier of the test statistic result. It represents the probability of getting as extreme or more extreme results under the assumption that the metric value doesn't change (hypothesis **H0** in the example). To learn more, see [this article](https://en.wikipedia.org/wiki/P-value). To accept a hypothesis, the **P-value** is compared with the significance level: **P-value** ≤ _alpha_. The default threshold is _alpha_ = 0.05. Keep in mind that _alpha_ sets the probability of type I errors (false-positives). It's impractical to set a very low alpha value, as it increases the probability of type II errors (false negatives) and raises the **MDE**. - **MDE** (%): Minimum detectable effect. This is the smallest change in the metric that can be detected with the existing amount of data and with the probability of type I errors _alpha_ = 0.05 and type II errors _beta_ = 0.2. The MDE is expressed as a percentage of a metric value in the control variant. You can lower the **MDE** by increasing the audience size and duration of the experiment. ### Other metrics {#additional-metric} {% note info %} The option is available with [paid AppMetrica pricing plans](https://appmetrica.yandex.com/docs/en/common/pricing/ru-currency). {% endnote %} You can expand reports on ongoing or stopped experiments with additional metrics. This is useful if you selected an irrelevant metric when creating the experiment or if you need to analyze the experiment's impact on other metrics. 1. In AppMetrica, go to **Varioqub → A/B experiments**. 1. Under the name of the experiment, click **View result**. 1. In the menu that appears, click the selected metric and choose **Other metric**. A list of available metrics opens, excluding those you've already selected. Select the desired metric to see how it was affected by the experiment. ### Filters {#filters} If you selected multiple metrics when [creating an experiment](https://yandex.com/support/varioqub-app/en/web/create.md), you can view a chart for each of them. To do this, select a metric name from the dropdown list above the chart. By default, complete data for each metric is shown. You can filter the data displayed in the chart and table. Currently, the following filters are available: **Time period** - **Entire period** (selected by default): The entire period of the experiment. - **Exact period**: Choose specific dates for which you want to view data on the chart in more detail. To do this, select **Exact period** in the dropdown above the chart and then select the dates to show results for. After that, click **Apply**. {% note tip %} If you want to view the results for a specific day, select **Exact period** and double-click a date in the date selection window. After that, click **Apply**. {% endnote %} **Additional filters** - **Triggering experiment**: Only display results for events the experiment was triggered for. - **Only full days**: Show results for the days when the experiment lasted a full day.
Contact support