Experiments with campaigns
- Find out how different types of campaigns affect each other.
- Analyze how conversions on a site are related to ad impressions.
- Compare the effectiveness of automatic strategies and manual bid management.
Use experiments to test various hypotheses concerning your ad performance, combination of settings, campaign types, and even media plans.
Before starting an experiment
Formulate the hypothesis you want to test. It should affect a single parameter only. We recommend testing just one hypothesis per experiment.
You must have goals configured in Yandex.Metrica. Run an experiment if you have at least 200 conversions per month for the goals you are interested in. Otherwise, the results may be statistically insignificant.
Please note that you can't run experiments on ads for mobile apps.
How to run an experiment
- Step 1. Create an experiment in Yandex.Audience
- Log in to the service with the account you use for running an experiment in Yandex.Direct.
- On the Experiments tab, click Create experiment.
- Enter a name and select the Yandex.Metrica tag you want to use for your experiment.
- Specify the number of segments and their proportions in the audience. The audience will be randomly distributed across segments in these proportions. An audience can be divided into two or more segments, not necessarily evenly. For example, if you suspect that enabling a certain setting might dramatically increase your advertising expenses, you can divide the audience 70% to 30%.
For more information about how to create an experiment and work with it, see Yandex.Audience Help.
- Step 2. Set up an experiment in Yandex.Direct
- Prepare your campaigns for testing. Make as many copies of the original campaign as there are segments in the experiment. The copied campaigns must be stopped. Leave one campaign unchanged, and change a setting based on your hypothesis in each of the other campaigns. Leave all the other settings, ads, and budget the same.
- Add the Yandex.Metrica tags that you specified when creating the experiment to all the campaigns.
- Connect your campaigns to the experiment. On the campaign settings page, go to Experiments and click . The experiment setup window opens.
Select the necessary experiments. One campaign can be connected to no more than 5 experiments at the same time. Specify the audience segment you want to serve ads in this campaign to.Note.
Experimental segments limit targeting. For example, if you have set ad serving for women who are 18-25 years old and you target a campaign on a segment containing 20% of the audience, it will include 20% of women who are 18-25 years old from the entire audience.
- You can configure bid adjustments for the selected segment. To do this, click in the segment line and indicate the percentage of bid increase or decrease. All bids for the campaign will change by the specified percentage.
- Stop the source campaign and start the copied campaigns at the same time.
The experiment is running. Ads will be served based on your experiment settings. The experiment must run long enough to accumulate sufficient statistics for proper result analysis. Usually this takes at least a month.
For other examples of experiment setup, see Usage.
- Step 3. Track experiment statistics
You can evaluate the results of your experiment based on spending data from Report Wizard and conversions from Yandex.Metrica. This will provide added insight on which version gives you the best ROI. Other indicators, such as clicks, impressions, or CTR, are not taken into account.
To analyze statistics in Yandex.Direct and Yandex.Metrica, do the following:
- Choose identical periods.
- Make sure that the experiment has not been interrupted for the entire period selected.
- Filter the data by your test region.
- In Yandex.Direct
In Report Wizard, select Columns — Total expenditure (RUB). Then add the filtering criteria Experiments, and select the relevant experiment.
Spending data for each segment is in the Total expenditure (RUB) column.
- In Yandex.Metrica
To view the experiment statistics in Yandex.Metrica, use the standard report “Yandex.Direct, experiments”. Select the experiment you want to analyze. Select the segment with Yandex.Direct campaigns created for the experiment. Click Comparison mode and select the control group. You will see a report about the given experiment. Add your Goals under the experiment: for example, adding an item to cart.
For more information, see Yandex.Metrica Help.
Let's say that you received the following results from your experiment
Segment Segment % Expenditures, RUB Conversions A 50 100,000 200 B 50 100,000 500 Segment Segment % Expenditures, RUB Conversions A 50 100,000 200 B 50 100,000 500
Based on these data, we can draw the following conclusions: at the same cost, segment B resulted in 150% more conversions (± 29% accounting for error) than A, while the cost per conversion decreased by 60% (± 5%). The campaign targeted at segment B is more effective.
To analyze the experiment results, use Calculator.
Read more on how to make decisions based on statistics
- Step 4. Stop the experiment.
After you have analyzed the experiment statistics, finish it. Stop the tested campaigns all at once. In the settings window, disconnect the experiment from the successful campaign you would like to continue serving.
You can use experiments to run tests of any level of complexity.
- Adding display advertising
If you have a contextual campaign, you can check how adding display ads can affect your campaign and website indicators.
In Yandex.Audience, create an experiment and specify the percentage of users to be served display ads. In Yandex.Direct, create a display campaign and target it at the relevant segment of your experiment. Connect a contextual campaign to the experiment. Distribute the advertising budget between the campaigns and set the applicable restrictions.
- Mobile traffic
You can find out how increasing the mobile bid can affect your campaign.
In Yandex.Audience, create an experiment and indicate the percentage of users to increase bids on mobiles for. In Yandex.Direct, make two copies of the source campaign and connect them to the experiment. Use a bid adjustment to increase the bid by 100% for mobile devices in one of the campaigns. Target the source campaign copy and the test campaign at their applicable segments.
- Bid adjustments for segments
You can see how increasing the bid affects the number of conversions.
In Yandex.Audience, create an experiment and indicate the percentage of users to increase the bid for. In Yandex.Direct, make two copies of the source campaign and connect them to the experiment. Use a bid adjustment to increase the bid in one of them. You do not need to specify the segment in this case.