Results and statistics
AB Test measures every experiment on conversion and revenue and reports Bayesian results, so you can tell which variant actually wins — and apply it in one click.
What AB Test measures
For each variant, AB Test records three things as visitors move through the test:
- Visitors — the unique visitors assigned to the variant.
- Conversions — the visitors who completed a purchase.
- Revenue — the order value attributed to the variant, including orders placed through Shop Pay and third-party checkouts. See Revenue attribution.
From these it derives each variant's conversion rate and revenue per visitor, then compares the variants statistically.
Bayesian results
Instead of a pass/fail significance test, AB Test uses a Bayesian approach and reports a probability to be best for each variant — the chance that variant is the true winner given the data so far.
- A variant at 95% probability to be best is very likely the winner.
- Two variants near 50/50 means there is not yet enough evidence to separate them — keep the test running.
Probability to be best answers the question you actually care about — "which variant should I ship?" — more directly than a p-value, and it stays meaningful as the test accumulates data.
Per-variant metrics
Each variant shows its conversion rate and revenue alongside its probability to be best, so you can weigh both whether a variant wins and by how much. A variant can win on conversion rate while another wins on revenue per visitor — AB Test surfaces both so you decide on the metric that matters for the test.
Segment breakdowns
Results can be broken down by segment to see who each variant really wins with:
- Device (mobile, tablet, desktop)
- New vs returning visitor
- Country
- UTM parameter
- Referrer
- Cookie
Segments are for reading results; to limit who enters a test, use audience targeting.
How long to run a test
Let a test run until the leading variant reaches a high probability to be best on a meaningful number of visitors and conversions — not just a handful. Ending too early on thin data risks calling a winner that regresses later.
As a rule of thumb, wait for both a clear probability to be best and enough conversions per variant that the result stays stable across a few days of traffic.
Apply the winner
When a variant proves itself, apply it as the winner in one click. AB Test promotes the winning variant to all traffic so every visitor gets the better experience. Experiments also end automatically at their scheduled end date, and results stay available after a test ends for later reference.
Next steps
- Revenue attribution — how revenue is credited to variants, even through third-party checkout.
- Audience targeting — control who enters a test.
- Campaigns — group related experiments and report on them together.