To prove that a retention tool or campaign works, split your players into a control group that nobody contacts, a group handled the way your team handles players today, and a group handled by following the new approach. Run it for a month, compare average turnover per player, and agree the rules before you start. It is the only way I know to get an answer that belongs to your players and not to a vendor's slide. This is the plan we use in a Sellrise pilot, written so you can use it with any vendor, including none.
Why do you need a control group at all?
Because players change behaviour for many reasons at once: season, promotions in your product, a payment method that started working, a new game. If you roll out a tool on everybody and turnover rises, you do not know whether the tool did it. A control group lives through the same weeks as everyone else but does not get the treatment, so the difference between the groups isolates what you changed.
Without one, a vendor can claim credit for a good month and an operator can blame a vendor for a bad one. Neither is knowledge.
What are the three groups?
- Control. Players nobody contacts. Not your team, not the vendor, not an automatic journey.
- Your team as today. The retention team works this group with its own lists and its own offers.
- With the new approach. The team works this group by following the suggestions: who to contact, when, with which offer and through which channel.
The middle group is the one people leave out, and it is the one that keeps the comparison honest. If the new approach only beats the control group, you have shown that contacting players helps, which you probably knew. The question you are paying to answer is whether it beats what your own team already does.
How do you assign players to groups?
At random, from the same pool, before anything is sent. A few practical rules:
- Decide the eligibility rule first (for example, active players with a deposit or a bet in the month, which is how we define an active player).
- Remove test, service and bot accounts before you split.
- Exclude self-excluded players and those flagged for responsible gaming from every group that could be contacted. They stay out of outreach whatever the design says.
- Record the assignment and do not change it halfway.
If your team has strong views about particular VIPs, deal with that before the test, not during. A manager who quietly contacts a control player to be helpful has spoiled the experiment.
Which metric should you pick?
One, written down in advance. In our pilot it is average turnover per player over the month, counted for every player in the group whether or not they responded. It is simple, hard to game and tied to money.
You can track other numbers for explanation (deposits, return rate, bonus cost). Treat them as notes. If you decide the winner by whichever metric looks good afterwards, you will find a winner every time.
How big should the groups be, and how long should the test run?
Big enough to see a difference above the noise, and long enough to cover the behaviour you care about. I cannot give you a universal figure, because the right size depends on your base and the spread of your metric. We agree group sizes and the metric with the operator before the start. A month is our default for retention work, since shorter windows tend to reward a deposit that was pulled forward.
If your base is small, say so early. A small base may make a clean three-group test hard, and it is better to know that before you spend a month on it.
What do you do during the month?
- Run the groups as agreed, with no mid-test changes to offers or rules.
- Keep a log of anything unusual in the product or the market, such as an outage or a big promotion.
- Check each week that the control group has stayed untouched.
- Let the team decide and send. In our model the system suggests and a person sends, so nothing goes out on its own.
Also decide beforehand who will read the result and how. I prefer the analyst on your side to calculate it from your own data.
How do you read the result?
Write down average turnover per player in each of the three groups, the cost of the bonuses and channels used, and the difference. Then ask two questions. Is the difference larger than you would expect from random variation? And is it larger than the cost of getting it?
A flat result is a result. It might mean the tool does not help your players, or that your base is too small to tell, or that your team already does the job well. Each of those is worth knowing before you sign a longer contract. We do not name percentages in advance, because any figure we could quote would come from other players.
What should you ask a vendor who proposes a test?
- Who chooses the groups and who calculates the result?
- Is there a group that your own team works as it does now?
- What is the metric, and was it fixed before the test?
- What happens to the data and to the price if the result is flat?
- Does anything get sent without a person approving it?
A vendor who will not agree to this is asking you to pay for a promise. For related reading, see how to measure whether a bonus worked and how casino CRM is priced. To run the test with us, start a pilot, or open the demo first.