Experiments with campaigns
- Find out how different types of campaigns affect each other.
- Analyze how conversions on a site are related to ad impressions.
- Compare the effectiveness of automatic strategies and manual bid management.
Use experiments to test various hypotheses concerning your ad performance, combination of settings, campaign types, and even media plans.
Before starting an experiment
Formulate the hypothesis you want to test. It should affect a single parameter only. We recommend testing just one hypothesis per experiment.
Set up your goals. Run an experiment if you have at least 200 conversions per month for the goals you are interested in. Otherwise, the results may be statistically insignificant.
Please note that you can't run experiments on ads for mobile apps.
How to run an experiment
- Step 1. Create an experiment in Yandex Audience
-
- Log in to the service with the account you use for running an experiment in Yandex Direct.
- On the Experiments tab, click Create experiment.
- Enter a name and select the Yandex Metrica tag you want to use for your experiment.
- Specify the number of segments and their proportions in the audience. The audience will be randomly distributed across segments in these proportions. An audience can be divided into two or more segments, not necessarily evenly. For example, if you suspect that enabling a certain setting might dramatically increase your advertising expenses, you can divide the audience 70% to 30%.
For more information about how to create and use an experiment, see Yandex Audience Help.
- Step 2. Set up an experiment in Yandex Direct
-
- Prepare your campaigns for testing. Make as many copies of the original campaign as there are segments in the experiment. The copied campaigns must be stopped. Leave one campaign unchanged, and change a setting based on your hypothesis in each of the other campaigns. Leave all the other settings, ads, and budget the same.
- Add Yandex Metrica tags that you specified when creating the experiment to all the campaigns involved in the experiment.
Connect your campaigns to the experiment. In the campaign settings, click Advanced settings, switch to the Experiments section, click Add, and select the experiment from the list or create a new experiment.
One campaign can be connected to no more than 5 experiments at the same time. Specify the audience segment you want to serve ads in this campaign to.
Note.Experimental segments limit targeting. For example, if you have set ad serving for women who are 18-25 years old and you target a campaign on a segment containing 20% of the audience, it will include 20% of women who are 18-25 years old from the entire audience.
You can configure bid adjustments, CPC, CPA, and cost revenue ratio for the selected segment. To do this, enable Adjustments and specify the adjustment value.
- Stop the source campaign and start the copied campaigns at the same time.
The experiment is running. Ads will be served based on your experiment settings. The experiment must run long enough to accumulate sufficient statistics for proper result analysis. Usually this takes at least a month.
For other examples of experiment setup, see Usage.
- Step 3. Track experiment statistics
-
You can evaluate the results of your experiment based on spending data from Report Wizard and conversions from Yandex Metrica. This will provide added insight on which version gives you the best ROI. Other indicators, such as clicks, impressions, or CTR, are not taken into account.
To analyze statistics in Yandex Direct and Yandex Metrica, do the following:
- Choose identical periods.
- Make sure that the experiment has not been interrupted for the entire period selected.
- Filter the data by your test region.
- In Yandex Direct
-
In Report Wizard, select Columns — Total expenditure (RUB). Then add the filtering criteria Experiments, and select the relevant experiment.
Spending data for each segment is in the Total expenditure (RUB) column.
- In Yandex Metrica
-
To view the experiment statistics in Yandex Metrica, use the standard report “Yandex Direct, experiments”. Select the experiment you want to analyze. Select the segment with Yandex Direct campaigns created for the experiment. Click Comparison mode and select the control group. You will see a report about the given experiment. Add your Goals under the experiment: for example, adding an item to cart.
For more information, see Yandex Metrica Help.
Let's say that you received the following results from your experiment
Segment Segment % Expenditures, RUB Conversions A 50 100,000 200 B 50 100,000 500 Segment Segment % Expenditures, RUB Conversions A 50 100,000 200 B 50 100,000 500 Based on these data, we can draw the following conclusions: at the same cost, segment B resulted in 150% more conversions (± 29% accounting for error) than A, while the cost per conversion decreased by 60% (± 5%). The campaign targeted at segment B is more effective.
To analyze the experiment results, use Calculator.
Read more on how to make decisions based on statistics
- Step 4. Stop the experiment.
-
After you have analyzed the experiment statistics, finish it. Stop the tested campaigns all at once. In the settings window, unlink the experiment from the successful campaign you would like to continue serving. To do this, click the cross button.
Usage
You can use experiments to run tests of any level of complexity.
- Adding display advertising
-
If you're running a Text & Image ad campaign, you can explore how adding display ads might impact your campaign and website indicators.
In Yandex Audience, create an experiment and specify the percentage of users to be served display ads. In Yandex Direct, create a display campaign and target it at the relevant segment of your experiment. Add your campaign with Text & Image ads to the experiment. Distribute the advertising budget between the campaigns and set the applicable restrictions.
- Mobile traffic
-
You can find out how increasing the mobile bid can affect your campaign.
In Yandex Audience, create an experiment and specify the percentage of mobile users to increase bids for. In Yandex Direct, make two copies of the source campaign and connect them to the experiment. Use a bid adjustment to increase the bid by 100% for mobile devices in one of the campaigns. Target the source campaign copy and the test campaign at their applicable segments.
- Bid adjustments for segments
-
You can see how increasing the bid affects the number of conversions.
In Yandex Audience, create an experiment and specify the percentage of users to increase the bid for. In Yandex Direct, make two copies of the source campaign and connect them to the experiment. Use a bid adjustment to increase the bid in one of them. Leave both segments selected: this is equivalent to disabling targeting by segment.
