A/B testing is used to test variants in a controlled experiment, with the aim if deducing which variant is most successful based on a defined outcome. The objective is to test a number of variants against a control, to see which variant is most successful.

deltaDNA’s A/B Testing provides the functionality to test any number of variants against a control by changing a number of parameters for each variant and then measuring the impact of these changes. A parameter is a key/value pair which is used to change a variable in the game such as the difficulty of a mission or the layout of a screen. Any number of parameters can be defined and passed back through the A/B testing interface giving you endless capability to test and optimise your game.

Using the powerful segmentation tool, deltaDNA A/B testing allows you to target a specific group of users with a test giving you the ability to be precise in exactly which users are included in the experiment.

deltaDNA A/B Testing

A/B testing requires at least two variants, although there is no limit to the number of variants that can be set up in a test. One variant is always served no parameters and is defined as the control, the other variants are served modified parameters as defined by the test. Each variant is automatically set up to be of equal size to keep the test statistically accurate.

An example of a test is to modify the difficulty of mission 3. Currently mission 3 has a difficulty set to 100 and you want to test if reducing of increasing the difficulty will improve the completion rate of the mission. You would set up a parameter missionDifficulty which returns an integer. Within the game, at the start of mission 3 the mission difficulty parameter is set to 100 and there is the ability to over ride it with a call to deltaDNA. The control gets the default value and two variants are set up which pass back missionDifficulty:50 and missionDifficulty:150. Each variant is automatically set up with 33% of the users.

63

You then set a conversion event which is the event that must be achieved by the user for them to count as a successful conversion. This event can be any event, with success based on a set of parameters of the event meeting the required criteria. This allows a conversion to be based on a precise event criteria.

There is nothing to stop you using deltaDNA to test multiple things in a single test. You can send as many parameters as you require in each test, giving you the ability to test a range of prices or alter aspects of the game in different ways.

The system will allow you to run tests simultaneously, the same users will be included in both tests. If you are confident the tests will not interfere with each other this is a good approach to maximise the number of participants in each test. If you want to make sure users are only selected for one test at a time, use the exclusive check box on the Test Details page. This will keep users that are eligible for two exclusive tests to be selected for only one of the tests.

Managing A/B Tests

deltaDNA provides two views of the running test:

  • The A/B Test Calendar gives an overview of all running
  • The A/B Test Details provides the current status of the test and the results

The A/B Test Calendar view shows the details of all the tests as well as a calendar showing the tests as they will happen. The calendar is colour coded:

  • Tests which are completely separate and have not overlap are shown in green
  • Tests with an overlap and therefore likely to have some contamination are shown in amber
  • Tests that fully overlap are shown in red

64

On the top half of the A/B testing screen you can Create a New Test, Edit a test that has not been completed and view a test while it is running of after it has completed.

65

Once a test is complete you can view the results in the A/B Test Details page

66

This screen gives you information on the running test while it is in progress and provides the information on the results of the test once it is complete.

Once a test is complete deltaDNA will calculate the confidence and significance of the test and show you the conversion for each variant. You can also use the A/B testing dashboard to look at any chart on the dashboard split by the A/B test to see the impact the test had on KPI’s.

Instrumenting Your Game

For A/B Testing to work you must instrument the game to support the parameters being fed back through the engage call.

  • You must make sure the game is designed to accept the parameters and their is a mapping between the parameters being sent and those in the game
  • If you want to target specific versions of the game, this data needs to be passed back through the SDK so segmentation can be used to identify those users
  • Any resources that need to be swapped need to be pre built into the game ahead of the A/B testing, tests can only change things that are defined in the game

Next Steps

  1. See how to Create an A/B Test
  2. Read about the Best Practices in running A/B Tests
  3. Learn about advanced techniques on A/B testing and Segmentation
  4. Learn how to QA your A/B Tests