Experiments with campaigns

Experiments allow you to check which Yandex.Direct ad campaign settings are most effective for your business. For example, you can:
  • Find out how different types of campaigns affect each other.
  • Analyze how conversions on a site are related to ad impressions.
  • Compare the effectiveness of automatic strategies and manual bid management.

Use experiments to test various hypotheses concerning your ad performance, combination of settings, campaign types, and even media plans.

Before starting an experiment

  1. Formulate the hypothesis you want to test. It should affect a single parameter only. We recommend testing just one hypothesis per experiment.

  2. Make sure that you have access to experiments in Yandex.Audience, Yandex.Direct, and Yandex.Metrica. If you do not have access, contact your manager at Yandex.

  3. You must have goals configured in Yandex.Metrica. Run an experiment if you have at least 200 conversions per month for the goals you are interested in. Otherwise, the results may be statistically insignificant.

  4. Please note that you can't run experiments on ads for mobile apps or display campaigns with deals.

How to run an experiment

Step 1. Create an experiment in Yandex.Audience

On the Experiments tab, click Create experiment. Enter a name and select the Yandex.Metrica tag you want to use for your experiment. Specify the number of segments and their proportions in the audience. The audience will be randomly distributed across segments in these proportions.

An audience can be divided into two or more segments, not necessarily evenly. For example, if you suspect that enabling a certain setting might dramatically increase your advertising expenses, you can divide the audience 70% to 30%.

For more information about how to create an experiment and work with it, see the Help for Yandex.Audience.

Step 2. Set up an experiment in Yandex.Direct
  1. Prepare your campaigns for testing. Make as many copies of the original campaign as there are segments in the experiment. The copied campaigns must be stopped. Leave one campaign unchanged, and change a setting based on your hypothesis in each of the other campaigns. Leave all the other parameters, ads, and budget the same.
  2. Add the Yandex.Metrica tags that you specified when creating the experiment to all the campaigns.
  3. Connect your campaigns to the experiment. On the campaign parameters page, go to Experiments and click Active experimentation and targeting → Edit. The experiment setup window opens.

    Select the necessary experiments. One campaign can be connected to no more than 5 experiments at the same time. Specify the audience segment you want to serve ads in this campaign to.

    Note.

    Experimental segments limit targeting. For example, if you have set ad serving for women who are 18-25 years old and you target a campaign on a segment containing 20% of the audience, it will include 20% of women who are 18-25 years old from the entire audience.

  4. You can configure bid adjustments for the selected segment. To do this, click in the segment line and indicate the percentage of bid increase or decrease. All bids for the campaign will change by the specified percentage.
  5. Stop the source campaign and start the copied campaigns at the same time.

The experiment is running. Ads will be served based on your experiment settings. The experiment must run long enough to accumulate sufficient statistics for proper result analysis. Usually this takes at least a month.

For other examples of experiment setup, see Use cases.

Step 3. Track experiment statistics

You can evaluate the results of your experiment based on spending data from Report Wizard and conversions from Yandex.Metrica. This will provide added insight on which version gives you the best ROI. Other indicators, such as clicks, impressions, or CTR, are not taken into account.

To analyze statistics in Yandex.Direct and Yandex.Metrica, do the following:

  • Choose identical periods.
  • Make sure that the experiment has not been interrupted for the entire period selected.
  • Filter the data by your test region.
In Yandex.Direct

In Report Wizard, select ColumnsTotal expenditure (RUB). Then add the filtering criteria Experiments, and select the relevant experiment.

Spending data for each segment is in the Total expenditure (RUB) column.

In Yandex.Metrica

To view the experiment statistics in Yandex.Metrica, open Yandex.Audiences, and click Results next to the relevant tag. You will see a report about the given experiment. Click the Metrics button and select the Conversions tested under the experiment that you want to analyze (for example, checkouts on your site or adding an item to the shopping cart).

Let's say that you received the following results from your experiment

Segment Segment % Expenditures, RUB Conversions
A 50 100,000 200
B 50 100,000 500

Based on these data, we can draw the following conclusions: at the same cost, segment B resulted in 150% more conversions (± 29% accounting for error) than A, while the cost per conversion decreased by 60% (± 5%). The campaign targeted at segment B is more effective.

Read more on how to make decisions based on statistics

Step 4. Stop the experiment.

After you have analyzed the experiment statistics, finish it. Stop the tested campaigns all at once. In the settings window, disconnect the experiment from the successful campaign you would like to continue serving.

Ways to use

You can use experiments to run tests of any level of complexity.

Adding display advertising

If you have a contextual campaign, you can check how adding display ads can affect your campaign and website indicators.

In Yandex.Audience, create an experiment and specify the percentage of users to be served display ads. In Yandex.Direct, create a display campaign and target it at the relevant segment of your experiment. Connect a contextual campaign to the experiment. Distribute the advertising budget between the campaigns and set the applicable restrictions.

Mobile traffic

You can find out how increasing the mobile bid can affect your campaign.

In Yandex.Audience, create an experiment and indicate the percentage of users to increase bids on mobiles for. In Yandex.Direct, make two copies of the source campaign and connect them to the experiment. Use a bid adjustment to increase the bid by 100% for mobile devices in one of the campaigns. Target the source campaign copy and the test campaign at their applicable segments.

Bid adjustments for segments

You can see how increasing the bid affects the number of conversions.

In Yandex.Audience, create an experiment and indicate the percentage of users to increase the bid for. In Yandex.Direct, make two copies of the source campaign and connect them to the experiment. Use a bid adjustment to increase the bid in one of them. You do not need to specify the segment in this case.