A/B testing is used to test variants in a controlled experiment, with the aim of deducing which variant is most successful based on a defined outcome. The objective is to test a number of variants against a control, to see which variant is most successful.
deltaDNA’s A/B Testing provides the functionality to test any number of variants against a control by changing a value of messages, parameterized content or popup images for each variant and then measuring the impact of these changes. A parameter is a key/value pair which is used to change a variable in the game such as the difficulty of a mission or the layout of a screen. Any number of parameters can be defined and passed back through the A/B testing interface giving you endless capability to test and optimise your game.
Using the powerful segmentation tool along with data passed to deltaDNA at run-time, the A/B test can target a specific group of users with a test giving you the ability to be very precise in exactly which users are included in the experiment.
deltaDNA A/B Testing
A/B testing requires at least two variants, although there is no limit to the number of variants that can be set up in a test. One variant is always served no parameters and is defined as the control, the other variants are served modified parameters as defined by the test. Each variant is automatically set up to be of equal size to keep the test statistically accurate, but you can manually adjust the size of the variants as long as they all add up to 100%.
An example of a test is to modify the difficulty of mission 3. Currently mission 3 has a difficulty set to DEFAULT and you want to test if reducing the difficulty will improve the completion rate of the mission. You would set up a missionDifficulty parameter containing an enumerated string value of DEFAULT, EASY, MEDIUM or HARD. The game will the start mission 3 with the difficulty set to DEFAULT, but the A/B Test will be used override it with a call to deltaDNA. The control gets the default value and two variants are set up which pass back missionDifficulty:EASY and missionDifficulty:HARD respectively. Each variant is automatically set up with 33% of the users.
You then set a conversion event which is the event that must be achieved by the user for them to count as a successful conversion. This event can be any event, with success based on a set of parameters of the event meeting the required criteria. This allows a conversion to be based on a precise event criteria. In this example, looking for a missionCompleted event, with the missionID equal to 3 would indicate that the player had completed the mission.
When your A/B Test is configured you will be shown a confirmation screen be so you can check the details before starting the test.
There is nothing to stop you using deltaDNA to test multiple things in a single test. You can send as many parameters as you require in each test, giving you the ability to test a range of prices or alter aspects of the game in different ways.
Managing A/B Tests
deltaDNA provides two views of the running tests, that can be filtered by date, or the decision point they they are triggered on.
- The A/B Test Calendar shows an overview of campaigns on a timeline
- The A/B Test List simply shows a prioritised list of campaigns.
The A/B Test Calendar is useful for showing overlapping campaigns and scheduled events.
The A/B test list screen lets you Create, Duplicate, View, Pause / Start and Delete tests.
You can view the results of an A/B Test after it has been started.
This screen gives you information on the running test while it is in progress and provides the information on the results of the test once it is complete.
Once a test is complete deltaDNA will calculate the confidence and significance of the test and show you the conversion for each variant. You can also use the A/B testing dashboard to look at any chart on the dashboard split by the A/B test to see the impact the test had on KPI’s.
Instrumenting Your Game
For A/B Testing to work you must instrument the game to be able to react to the parameters being returned to it.