Stop Counting A/B Test Wins. Audit What Your Experiments Teach You

A full experiment calendar is not evidence of a strong growth program. If six tests adjust CTA wording while nobody tests whether customers want the offer behind it, the team may be getting faster at answering the wrong question. Audit the problems your tests address, whether the results support the claims made about them, and what changed afterward.

A busy test calendar can conceal one unanswered problem

Test count measures throughput. Win rate measures how often a team labels a treatment successful. Neither tells you whether the team reduced an important uncertainty or avoided an expensive mistake.

Small tests have a place. A checkout team may already know that customers understand the offer and need to choose between two clear ways to present it. In that case, copy or layout changes can be worth testing. The warning sign is repetition without a new decision: another button label, then another headline, each addressing the same unclear value proposition while the underlying offer goes unexamined.

The opposite of a win can also be valuable. A well-measured test showing that a proposed change harms conversion can prevent a costly rollout. A precise result showing that any plausible benefit is too small to justify implementation can free up the roadmap. But a conclusive answer to a low-value question may still have been a poor use of traffic.

Speero suggests reviewing the share of completed tests that reach a conclusive answer for each problem statement, counting clear losses as well as wins. It calls this a “learning rate,” while noting that the definition is its own. Use that share as a prompt for investigation, not as a universal score. It does not measure the importance of the problem, the quality of the insight or whether anyone acted on it.

Group tests by customer problem before judging results

Start the audit one level above the variant. Give every test a stable problem-statement ID and write the customer uncertainty in plain language. For example, an illustrative P-01 might be: “Qualified visitors reach the pricing page but do not understand what is included in the entry offer.” Attach the customer evidence that prompted the question. Then record the specific hypothesis for each proposed treatment.

That structure exposes a common portfolio problem. Three CTA-copy tests and a headline test might all belong to P-01. They are four tests, but not four independent bets about customer needs. A test that changes the contents of the offer still addresses P-01, yet probes a different explanation for the same observed behavior. If copy changes repeatedly fail and the offer itself remains untouched, the next useful move may be research or an offer redesign rather than another synonym for “Get started.”

Do not assume the offer test will explain why customers respond differently. A quantitative experiment can help establish the effect of a change on measured outcomes; interviews and other qualitative work can investigate the mechanism. That distinction matters when deciding what to build next. Marty Cagan’s account of experimentation separates optimization of an existing experience from discovery that informs what to build, and distinguishes the roles of qualitative and quantitative evidence.

At the portfolio level, ask which important problems received no credible test, which received repeated minor variations and which cannot sensibly be resolved by an A/B test. Low-traffic, long-latency and new-product questions may need prototypes or interviews first. Those methods address different uncertainties; they are not substitutes for evidence of causal lift. Rank problems by potential business impact, customer importance and traffic cost, not by how easily they produce conclusive scorecards.

Check whether each result supports the conclusion attached to it

Before reviewing outcomes, inspect the plan. What was the falsifiable hypothesis? Who was actually affected by the treatment? Which primary metric would evaluate it, what effect was worth acting on, and how much sample and runtime were planned? Microsoft’s pre-experiment guidance calls for a falsifiable hypothesis, suitable metrics and a power calculation based on the users the treatment affects. Without those decisions up front, a team can too easily choose the most flattering metric after seeing the results.

At readout, keep the estimate and its interval beside the decision. Use distinct labels:

• Supported improvement: the evidence supports an actionable benefit on the planned outcome, subject to guardrails.

• Supported harm: the evidence is strong enough to reject the treatment on a relevant outcome.

• Decision-relevant null: the interval is precise enough to rule out a benefit large enough to justify the change. This does not establish that the true effect is exactly zero.

• Inconclusive: the interval still includes effects that would lead to different decisions. A positive point estimate is not a win if the interval also supports no improvement or harm.

• Invalid: assignment, exposure, events or another measurement fault prevents a trustworthy readout.

A narrow interval below zero can justify rejecting a treatment. It does not, by itself, reveal which belief about customers was wrong. Likewise, an early safety shutdown should not be filed as an ordinary ambiguous result or quietly removed from the audit. Record it separately, along with its guardrail signal and follow-up. Microsoft distinguishes monitoring guardrails for harm from repeatedly looking at data to declare a winner; statistical claims need decision rules that account for those looks.

Then audit the technical chain. Check expected allocation against observed assignment counts; verify that the intended audience was exposed; inspect conversion events across the full journey, including domain boundaries and consent limitations; and compare performance and commercial guardrails by variant. Microsoft checks sample-ratio mismatch before interpreting effects. In one analysis, roughly 6% of its tests had an SRM, and an MSN example showed bot filtering reversing the apparent result. That percentage is not a benchmark for your program. The lesson is that a plausible-looking scorecard can describe the measurement pipeline rather than customer behavior.

For React and headless sites, assignment, exposure and variant-level QA deserve explicit checks. The same applies to reliable analytics and conversion events. If the audit identifies broken assignment, tracking or performance guardrails, growth engineering support can help repair that implementation; product leaders still own the customer question, and the team needs an agreed statistical policy.

Trace what changed after the readout

An experiment record should not end with “won” or “lost.” Add a field called “changed what we did next,” with a link to the shipped release, rejected proposal, roadmap change or new research question. If no decision changed, record that too. This is an operational audit field, not another invented learning-rate benchmark.

Consider the illustrative CTA tests. Suppose their results are inconclusive, while a later offer test gives enough evidence to reject the proposed offer change. The first finding may justify repairing measurement or narrowing the question; the second may save an implementation. Neither becomes useful to the next team if the archive contains only a green or red badge.

Store the original problem, hypothesis, affected audience, primary metric, effect estimate and interval, limitations, guardrail movements and ship decision together. Microsoft recommends archiving hypotheses, metadata, metric movements and decisions so later teams can revisit ideas and understand shipped changes. This also prevents a local funnel metric from settling a broader product decision by accident: in a Microsoft Power Automate example, easier Copilot entry substantially increased entry, while the downstream save-rate proxy was lower for the treatment.

Run the quarterly review as a decision meeting

Bring a register grouped by problem-statement ID, not a slideshow of winning variants. For each problem, show the customer evidence, tests attempted, traffic spent, outcome labels, measurement defects, unresolved uncertainty and decision taken. Ask the owner what the next quarter will build, research, retest or stop. If the answer is “another similar variant,” require a reason the previous attempts did not resolve the question.

Do not optimize this meeting for the highest possible share of individually conclusive tests. Netflix’s return-aware experimentation discussion treats traffic and implementation effort as constrained resources; under particular cost assumptions, more smaller tests can improve expected portfolio returns. It also warns that selecting a winning arm and estimating its return from the same data creates winner’s-curse bias. Precision, expected value and traffic cost must be considered together.

Quarterly audit checklist

1. Export every test. Include invalid runs and safety shutdowns. Assign a problem ID, owner, customer evidence and intended decision.

2. Recover the plan. Record the predeclared primary metric, smallest effect worth acting on, affected audience, sample and runtime plan, guardrails and stop rule. Mark missing plans as missing rather than reconstructing them from the result.

3. Verify measurement. Check assignment versus expected allocation, exposure, conversion events, cross-domain paths, consent limitations and variant-specific performance before scoring outcomes.

4. Relabel readouts. Separate supported improvement, supported harm, decision-relevant null, inconclusive and invalid. Keep the estimate and interval visible.

5. Follow the decision. Link each finding to a release, rejected change, research task or roadmap decision. Flag results that produced no recorded action.

6. Choose the next problems. Weigh customer importance and expected value against evidence quality and traffic cost. Fund measurement repairs separately from new tests, and stop testing surfaces that no longer address the main uncertainty.