Results and statistics
Sofcy - A/B Test & CRO measures every experiment on conversion and revenue and reports Bayesian results, so you can tell which variant actually wins — and apply it with the right action for the test type.
What Sofcy - A/B Test & CRO measures
For each variant, Sofcy - A/B Test & CRO records three things as visitors move through the test:
- Visitors — the unique visitors assigned to the variant.
- Conversions — the visitors who completed a purchase.
- Revenue — the order value attributed to the variant, including orders placed through Shop Pay and third-party checkouts. See Revenue attribution.
From these it derives each variant's conversion rate and revenue per visitor, then compares the variants statistically.
Bayesian results
Instead of a pass/fail significance test, Sofcy - A/B Test & CRO uses a Bayesian approach and reports each challenger's chance to beat control — the probability that the challenger's true conversion rate is higher than the control's, given the data so far. (Control is the baseline, so this is shown for the challengers, not for control itself.)
- A challenger at 95% chance to beat control is very likely the winner.
- A challenger near 50% means there is not yet enough evidence to separate it from control — keep the test running.
Chance to beat control answers the question you actually care about — "is this challenger better than what I have?" — more directly than a p-value, and it stays meaningful as the test accumulates data.
Per-variant metrics
Each variant shows its conversion rate and revenue, and each challenger shows its chance to beat control, so you can weigh both whether a variant wins and by how much. A variant can win on conversion rate while another wins on revenue per visitor — Sofcy - A/B Test & CRO surfaces both so you decide on the metric that matters for the test.
Segment breakdowns
Results can be broken down by segment to see who each variant really wins with:
- All traffic
- Device — Mobile (includes tablet) or Desktop
- New vs. returning visitor
- Marketing channel — Direct, Organic search, Paid search, Organic social, Paid social, Email, Referral, or Other
Segments are for reading results; to limit who enters a test, use audience targeting.
How long to run a test
Let a test run until the leading challenger reaches a high chance to beat control on a meaningful number of visitors and conversions — not just a handful. Ending too early on thin data risks calling a winner that regresses later.
As a rule of thumb, wait for both a clear chance to beat control and enough conversions per variant that the result stays stable across a few days of traffic.
Apply the winner
When a variant proves itself, mark it the winner from the results page. What "apply" does depends on the test type:
- URL tests — Sofcy - A/B Test & CRO creates a Shopify URL redirect from the control page to the winning page, so all traffic lands on the winner.
- Template and theme tests — applying ends the test and tells you exactly which template file or theme to make your default; you promote it in the theme editor (the app never edits your themes).
- JavaScript API tests — there is no apply step: you already own the rendering, so keep the winning variant's code and remove the losing branch.
If you set a scheduled end date, the experiment ends automatically when it arrives; otherwise it runs until you end it. Results stay available after a test ends for later reference.
Next steps
- Revenue attribution — how revenue is credited to variants, even through third-party checkout.
- Audience targeting — control who enters a test.
- Campaigns — group related experiments and report on them together.