Cohort Analysis Example:
The promotion that was run was a success and the developer was keen to understand in detail how the promotion impacted player behaviour. Was the promotion changing when players spent money and how players interacted with the game.
This type of analysis is vital to understand the impact of promotions on the playing base and how specific promotions can change behaviour and encourage monetisation and retention.
To undertake this analysis it was decided to look at two cohorts of players. The first cohort would be new players that were exposed to the promotion and the second would be a cohort of new players that didn’t get exposed to the promotion. This would allow analysis of the impact of the promotion on the playing behaviour.
As there was no A/B testing set up on the promotion, the solution was to use two date ranges to simulate the A/B test.
Using the Cohort Comparison tool makes this type of analysis easy and intuitive.
Cohort 1
The first Cohort is the promotion Cohort and will look at new players who joined the game on the day of the promotion and analyse their behaviour between the 5th of January and the 19th of January. This is 2,610 players with 105 paying players (4.02% paying)
- Select date range 2013-01-05 -> 2013-01-19
- Select user start date 2013-01-05
Cohort 2
The second Cohort is the non promotion Cohort and will look at new players who joined the game after the promotion on the 15th of January and analyse their behaviour between the 15th of January and the 29th of January. This is 1,707 players with 45 paying players (2.63% paying)
- Select date range 2013-01-15 -> 2013-01-29
- Select user start date 2013-01-15
In column 1 select Daily, Weekly and Monthly Active Users and in column 2 select Total Revenue by Time to First Payment.
The screen should look like this:
In Cohort 1 the active user chart shows the 2,610 users on the 6th. By the 20th there are 232 players still playing (8.8% 14 day retention).
In Cohort 2 the active user chart shows the 1,707 users on the 15th. By the 29th there are 140 players still playing (8.2% 14 day retention).
Time to First Payment shows how quickly players will start to spend money. Does the promotion encourage players to spend money earlier?
Because of the promotion, users are highly motivated to spend early. 90% of the players in Cohort 1 are spending within the first hour, compared to Cohort 2 where the majority are spending later in their gameplay.
Looking at the revenue, Cohort 1 generated $4,660 over the 14 days with the majority of that revenue in the first 6 days. Cohort 2 generated only $2,434 with a more even distribution of revenue across the two weeks.
If we look at the break down of spending players by their value we can start to get a better feeling for who is spending the money in each Cohort. Cohort 1 consists of low value players with 23% of the revenue generated by players spending $21-$50. This is in marked contrast to Cohort 2 which has a much higher value user base, with 44% of the players spending $101-$500.
Finally look at Revenue by Player Current Value and One, Seven, Fourteen & Thirty Day Retention.
Looking at the revenue by player current value on a time line over the date range we see a different picture. Cohort 1 has a lot of high value whales early, but they drop out towards the end of the period and the likelihood is those players have left the game. Cohort 2 shows a different picture with high value players increasing over the time period, although at an overall lower level than Cohort A.
Finally taking a look at the retention of both groups, Cohort 1 overall shows a worse retention rate compared to Cohort 2.
There is clearly a difference in playing behaviour for new players when a promotion is used. This is not simply encouraging the existing user base to spend money, but it is also about changing the behaviour of new players. New players are spending much earlier in the gameplay experience and that spending is continuing over the lifespan of the players. The players in the promotion are spending smaller amounts, with the promotion cohort generating less whales. The promotion has therefore generated an initial spike but this has been at the detriment of long time high value players.
On retention there appear to be a negative impact of the promotion on new players, retention rate, with cohort A showing showing worse retention rates for 1 and 7 day retention although the long term retention appears very similar.
This underlines the value of promotions to encourage new players to start spending and increase monetisation. There does seem to be an impact on retention and life time value and further analysis would need to be done to look at this impact in more detail.
