A/B testing
split testingA/B testing means comparing two versions by splitting the audience at random: half sees version A, half version B, and the data says which produces more. The method replaces opinions with measurement on pages, in ads and in emails.
Good to know
Testing’s value compounds: a single test rarely produces a miracle, but a rhythm of ten tests a year where every third wins grows conversion by tens of percent. The biggest finds come from big changes: the offer, the headline and the page layout before the button colour.
Statistical reliability is the bedrock: a result is declared only when the sample suffices and significance is reached, typically at the 95 % level. With small traffic, aim tests at pages with volume or accept longer test periods.
IN PRACTICE
How a test runs
A hypothesis from data
Analytics or a heatmap points to a problem, and the hypothesis claims: change X lifts metric Y, because Z.
One variable at a time
Versions differ in one thing so the result is interpretable. Several changes need a multivariate test and multiples of the traffic.
Winner shipped, lesson logged
The winning version goes live for everyone and the result enters the test log. Even a lost test is a purchased lesson.
PITFALLS
Two testing mistakes
The test is stopped as soon as a leader shows
Early swings usually even out. A pre-decided sample size and duration protect against false winners.
Button colours get tested, the offer does not
Small cosmetic tests produce small results. The biggest lever sits in the value proposition, pricing and headline.
FREQUENTLY ASKED
Frequently asked questions
How long should an A/B test run?+
At least one full week for weekly rhythm, and until the sample suffices, typically 1–4 weeks. A hundred conversions per version is a rough minimum.
Can I test with little traffic?+
Yes, within limits: test high-impact changes on your most visited pages, or in ads where the platform handles the split. Micro-change testing needs volume.
What tools does testing need?+
Ad platforms have testing built in, as do email tools. Site testing has dedicated tools with free tiers.
What should I test first?+
The headline and call to action of your most important page: biggest traffic times biggest impact. The queue is prioritised by impact over effort.
Why did the winner not lift the overall numbers?+
The test won with a certain audience in a certain context; overall numbers also carry season and traffic mix. That is why lessons are confirmed with a follow-up period.
FOR YOUR BUSINESS
Want results, not just definitions?
We go through your goals and propose channels and a budget in 30 minutes, free of charge.
FOR STUDENTS AND NEWCOMERS
Learn the terms, then come and put them to work
The glossary is free study material. Go deeper in our training, and if you want to work in the field, our growing team is looking for people.
Maintained by the VSCY editorial team · Updated 6 Aug 2026 · Part of the marketing glossary (99 terms)
