You measure a bonus by comparing the players who received it with a similar group who did not, over the same period, using the same metric. Anything else counts deposits that would have happened anyway. The number that matters is the difference between the two groups, and it is usually smaller than the raw count of deposits after the offer. I will walk through how to set it up and what tends to go wrong.
Why are deposits after a bonus a misleading number?
Because some of those players were going to deposit regardless. A bonus sent to active players on a Thursday will be followed by deposits, since active players deposit. If you credit all of them to the bonus, every campaign looks successful.
The question to ask is what would have happened without it. You cannot observe that for the players who got the bonus, so you observe it for a group who did not. That group is the control group, defined in our glossary as players nobody contacts, held aside so the real effect can be measured against them.
How do you build a proper control group?
Four rules matter more than the rest:
- Decide the group before the campaign. Not after you see who responded.
- Select at random from the players who qualify for the offer, so the two groups look alike.
- Keep the control group untouched. If other teams or channels contact them, the comparison is spoiled. Tell them.
- Make it big enough to see a difference. A group of a few dozen players tells you very little. Agree sizes with your analyst up front, since the right size depends on how noisy your metric is.
I cannot give you a universal size, because it depends on your base and on how large an effect you expect. In a Sellrise pilot we agree group sizes and the metric before we start, so nobody can move the goalposts later.
Which metric should you use?
One metric, chosen in advance. I prefer average turnover per player over the test period, per player in the group, including players who did nothing. That last part matters. If you only count players who responded, you have thrown away the information about the ones who did not.
Other metrics are useful as context. Deposit conversion, return rate and bonus cost tell you how the result happened. But if you pick the flattering one after the fact, you will find a winner in almost any campaign. The metric and the period belong in a written plan.
Watch the cost side as well. A bonus that raises turnover but costs more than the extra margin it brought is not a success. Margins, wagering terms and limits are yours to model. A forecast tool cannot know them.
How long should you run the test?
Long enough to include the behaviour you care about. For a short promotion that may be a few weeks. For a retention programme, we use one month, with average turnover per player as the metric. A bonus can pull a deposit forward and leave a hole afterwards, so a measurement that stops the day after the offer will overstate it.
This is also why I would compare periods of the same length, with the same calendar features. A holiday weekend in one group's window and not in the other's would swamp the effect.
What is a three-group design, and why use it?
A simple test has two groups: contacted and not contacted. A more useful one for a retention team has three: a control group that nobody contacts, your team working as it does today with its own lists and offers, and your team following the suggestions of a system. After one month you compare average turnover per player in each. The difference between the control and the others is the effect of contact at all, and the difference between your team as it is and your team with suggestions is the effect of the system.
This is the design we run in a Sellrise pilot. It keeps the vendor honest, because the baseline is your own team and not an empty chair. The method is described step by step in how to run a control group test for casino retention.
What mistakes flatter campaigns?
- Counting every deposit in a window after the message, without a control.
- Choosing the comparison group after the campaign, from players who look similar.
- Measuring only responders.
- Ignoring the later dip that follows a pulled-forward deposit.
- Comparing a campaign to last month, when the base, the season and the promotions were different.
- Changing the definition of an active player between periods.
Each of these can make a weak campaign look strong. I have no wish to make anyone's work look worse, but a decision to spend more on a campaign should rest on a comparison that could have shown it did nothing.
What does a good result look like in practice?
It is a small table: the average turnover per player in each group, the cost of the bonuses given, and a note on how sure you are. Sometimes it is a positive effect, and you scale. Sometimes it is flat, and you have found money you can save. Both are useful.
I will not quote a typical percentage. Any number I could give would come from other operators, other markets and other offers, and your base differs. We do not name percentages before a test. You get the number on your own players.
How can you try this on your own base?
Start with one recurring campaign and hold out a random group. If you want the per-player version, Sellrise shows what bonuses actually did: deposits after the bonus, and how much of that the bonus itself brought. See the demo for the layout, start a pilot to measure on your players, and read who should not get a casino bonus for the decision side.