Is your test's traffic split broken?
If your groups are more uneven than chance allows, something in the test is broken and the result can't be trusted. Enter the users in each group to check.
Sample ratio mismatch: don't trust this test yet
With the split you planned, groups this uneven would happen by chance less than 1 time in 1,000 (p = <0.0001). Something is dropping or double-counting users in one group.
| Group | Observed | Expected | Difference |
|---|---|---|---|
| Control | 50,31250.67% of users | 49,645 | +1.34% |
| New paywall | 48,97749.33% of users | 49,645 | -1.34% |
Chi-square goodness of fit: χ² = 17.95, 1 degree of freedom. Flagged when p < 0.001. How this works
Questions people ask
What is a sample ratio mismatch?
It's when the number of users in each group is further from your planned split than random chance can explain. A 50/50 test with 50,312 users in one group and 48,977 in the other looks close, but with groups that size the gap is very unlikely to be luck.
What usually causes it?
The paywall view is logged only after something that differs between variants, like a slower image or a crash, so one group loses events. Other causes: assigning users after a condition the variant changes, bots or internal devices filtered in one group only, a variant that started later, or changing the split during the test.
Why flag at p < 0.001 instead of 0.05?
Mismatch checks are usually run every day across many tests. A strict cutoff keeps false alarms rare, and real mismatches from bugs nearly always produce tiny p-values anyway.
Can I just reweight the groups?
No. The mismatch tells you that the people in one group are different, not just fewer. Find the cause, fix it, and restart the test.
Get your test results in your inbox every Monday
We're building a weekly report for subscription apps. Connect RevenueCat or upload an export, and each Monday you get every running paywall test's revenue result, a check that the split is healthy, and what to test next.
Planned price: $49 a month, or $99 for teams.