A/B Testing for Beginners: Run Your First Test
Most marketing arguments end with "I think this headline is better." A/B testing ends them differently: it shows you two versions to real visitors, measures which one actually gets more clicks or sign-ups, and lets the data decide. It replaces opinion with evidence — and once you've run one honest test, you rarely go back.
This guide covers the whole beginner's arc: what A/B testing is, the scientific method behind it, what's worth testing (and what isn't yet), and the two ideas that trip up almost every newcomer — sample size and statistical significance. By the end you'll be able to plan a test that gives you an answer you can trust rather than a number you can fool yourself with.
- A test beats an opinion. A/B testing shows two versions to comparable audiences and measures which converts better.
- It's the scientific method. Form a hypothesis, change one thing, measure the result against a control.
- Sample size decides everything. Too little traffic and any "winner" is likely just random noise.
- Don't stop early. Peeking and calling a winner the moment it looks good is the most common way to get fooled.
- Test big, obvious things first — headlines, offers, calls to action — not button-shade tweaks.
What A/B testing actually is
An A/B test (also called a split test) is a controlled experiment. You take one page or element, create a second version with a single deliberate change, and split incoming traffic randomly between them. Version A is the control (what you have now); version B is the variant (your idea). Because visitors are assigned at random and both versions run at the same time, the only systematic difference between the two groups is the change you made — so any reliable difference in conversion rate can be attributed to it.
It's the same logic as a clinical drug trial: a treatment group, a control group, random assignment, and a measured outcome. Google's experimentation guidance frames experiments exactly this way — as controlled comparisons rather than before-and-after guesses. (See Google's experiments and Optimize documentation.)
The scientific method, applied
A good test isn't "let's try a new page and see." It's a structured loop you can repeat forever. The four steps:
- Hypothesis. A specific, falsifiable prediction: "Changing the headline to lead with the outcome will increase sign-ups, because visitors currently bounce before understanding the benefit."
- Variant. Build version B with exactly one meaningful change from the control.
- Measure. Pick one primary metric before you start (e.g. sign-up rate) and run until you have enough data.
- Decide. Keep the winner only if the result is statistically significant; otherwise treat it as inconclusive and move on.
Writing the hypothesis first is what separates an experiment from a guess. It forces you to name what you expect and why — so a losing result still teaches you something about your audience.
State your primary metric and how long you'll run the test before you launch. Deciding what counts as success after you've seen the numbers is how honest people accidentally fool themselves.
What to test (and what to skip early)
Beginners waste tests on tiny changes. With limited traffic, only big swings produce a difference large enough to detect. Focus your first tests on the elements that most influence whether someone acts.
Headlines are the highest-leverage test on most pages — they're the first thing read and set the entire value proposition. Try a benefit-led headline against a feature-led one. See landing page copywriting for how to draft variants worth testing.
Calls to action — the button copy and its wording move real numbers. "Start my free trial" vs "Sign up" changes how committed the click feels. Test the message, not just the color.
Images — a hero photo of the product in use vs an abstract graphic, or a real person vs a stock illustration, can shift trust and attention. Big visual swaps are easier to detect than subtle ones.
Layout & offer — form length, the order of sections, or the offer itself ("14-day trial" vs "money-back guarantee"). Offer changes often produce the largest lifts because they change the deal, not just the wording.
What to skip early: button-shade tweaks, one-word microcopy edits, and font changes. They can matter at massive scale, but on a small site the effect is usually too small to measure before your traffic runs out. Test things you'd expect to move the needle by a meaningful margin.
Sample size and statistical significance
This is the concept that makes A/B testing trustworthy — and the one most beginners get wrong. When you compare two conversion rates, some difference will always appear by pure chance, the same way flipping two coins 20 times rarely gives an identical count of heads. Statistical significance is the answer to: "how likely is it that a difference this big happened by random luck rather than because B is genuinely better?"
The industry convention, borrowed from science, is 95% confidence — meaning you accept a result only when there's roughly a 5% or smaller probability the observed difference is a fluke. This is a convention, not a law of nature; higher-stakes decisions sometimes demand more. Optimizely and CXL both walk through this reasoning in their statistics primers. (See CXL's guide to A/B testing statistics and Optimizely's statistical significance glossary.)
Reaching 95% confidence requires enough visitors and conversions — the sample size. The smaller the real difference between A and B, the more traffic you need to detect it reliably. That's why big changes on high-traffic pages resolve quickly, while a tiny tweak on a low-traffic page may never reach significance. Evan Miller's sample-size calculator estimates the visitors needed before you start, from your current rate and the smallest lift worth detecting.
Calculate the sample size you need before the test, not after. A test with too little traffic can't give a trustworthy answer no matter how the numbers look.
How long to run — and why stopping early is dangerous
The most damaging beginner habit is peeking: watching the dashboard and declaring a winner the moment B pulls ahead. Early in a test, conversion rates swing wildly — a handful of conversions can flip who's "winning." Stop the instant a version looks good and you're not measuring the better variant; you're catching a random high point. Do it repeatedly and you'll "find" winners that vanish once they go live.
To avoid it, commit up front to a fixed sample size or run length and don't decide until you reach it. A practical rule: run for at least one to two full weeks so the test spans weekday, weekend and full purchase cycles, and reach your pre-calculated sample size — whichever comes later.
Why does peeking inflate false positives?
Every time you check an in-progress test and let yourself stop, you get another chance to catch a random fluctuation that crosses the significance line. More looks means more chances for luck to trick you, so the real error rate climbs well above the 5% you thought you accepted. Fixing the endpoint in advance removes those extra chances.
One variable at a time, or many at once?
Start with one change per test (classic A/B). If B beats A, you know exactly why. Multivariate testing changes several elements at once to study how they interact — but it splits your traffic across many combinations, so it needs far more visitors to reach significance. Save it for high-traffic pages after you've mastered single-variable tests.
What if the test ends inconclusive?
That's a valid, common outcome — it means the change didn't move your metric enough to detect. Don't force a winner. Keep the control, record what you learned, and test a bigger, bolder change next. A flat result still narrows down what your audience does and doesn't care about.
Common pitfalls to avoid
Most failed tests fail for a handful of predictable reasons. Watch for these:
- Peeking and early stopping — the big one, covered above.
- Too little traffic — running a test on a page that gets a trickle of visitors, so it can never reach significance.
- Testing trivial changes — micro-tweaks whose true effect is smaller than the noise you can measure.
- Changing more than one thing in a supposed A/B test, so you can't tell which change caused the result.
- Ignoring the weekly cycle — ending a test after two good days that happened to be a weekend.
- Chasing the wrong metric — optimizing clicks when revenue is what you care about. See marketing metrics that matter to pick the right one.
Use this checklist to plan an experiment before you launch it — progress is saved in this browser:
- I wrote a specific, falsifiable hypothesis and the reason behind it
- Version B changes exactly ONE meaningful element from the control
- I chose one primary metric before starting
- I estimated the sample size needed to detect a worthwhile lift
- I set a fixed end point (sample size and/or run length) in advance
- The test will run at least one to two full weeks
- I committed to NOT stopping early no matter how the numbers look
Common questions
Do I need special software to run an A/B test?
You need a way to split traffic and measure the outcome. Dedicated experiment platforms handle randomization and significance math for you, and analytics tools let you tie results to conversions. You can run simple tests manually, but a proper tool prevents mistakes in assignment and measurement. Whatever you use, pair it with solid analytics — see our Google Analytics 4 basics guide.
How big a difference should I expect?
It varies enormously and there's no reliable "typical" number to promise. Many tests produce small or no change; the occasional big win comes from changing the offer or fixing a broken page. Plan for modest, compounding improvements — and never assume a figure from a case study will replicate on your site.
Can I A/B test emails too?
Yes — subject lines, send times, and content are all testable, and most email platforms have split testing built in. The same rules apply: one variable, a pre-set metric, and enough recipients to reach significance. Our guide to email automation and drip campaigns covers where testing fits into a sequence.
Where to go next
A/B testing is the engine of optimization, but it needs a program around it to pay off. Read our guide to conversion rate optimization basics to see how tests fit into a repeatable improvement loop, and marketing metrics that matter to make sure you're testing against a number that actually reflects business results.