Stop Reserving Separate Traffic for Every A/B Test

Reserve separate traffic for A/B tests only when sharing users would make the experience unsafe or the decision hard to interpret. Two independently assigned experiments can often run together. Isolation has a real cost: it reduces the sample available to each test. The better rule is to overlap by default, review plausible collisions before launch, and design explicitly for interactions that could change what you ship.

The traffic-isolation rule is costing you answers

Suppose a growth team wants to test a product-page change and a checkout change. One option is to give each experiment half the eligible audience, ensuring no user enters both. The other is to assign users independently, allowing both tests to use the full eligible audience. Isolation feels safer, but it can make both answers arrive much later.

A Speero traffic illustration assumes a 3% baseline conversion rate, a 10% relative lift, 80% power, 95% confidence, and about 16,000 eligible visits per month. Its approximation calls for roughly 51,700 visitors per variant. With the full audience, that is about 6.5 months for one 50/50 test. Split the audience equally between two concurrent tests, and each needs about 13 months to reach the same target. Those figures describe the illustration, not your site’s forecast; calculate against your own eligible users and outcome.

The cost has two useful interpretations. At a fixed end date, halving each test’s sample raises an approximately inverse-square-root-of-sample minimum detectable effect by √2, or about 41%. If the original detectable effect matters, retaining it takes roughly twice as long under the same assumptions. Neither shortcut replaces a power calculation for the actual design. But they make the trade-off visible: isolation is a decision to accept less sensitivity or wait longer, not a free precaution.

That cost is worth paying when the combined experience could break, contaminate eligibility, or make a consequential shipping decision unreliable. It is not worth paying merely because two test names appear on the same launch calendar.

Overlap is not automatically interference

With independent 50/50 assignments to two binary tests, users can land in four combinations: neither treatment, A only, B only, or both. For B, the interaction question is whether B’s effect among users assigned to A’s treatment differs from B’s effect among users assigned to A’s control. Statsig’s interaction documentation describes that contrast as a difference of differences across the four cells and reports cell counts and overlapping unique users.

Independent randomization generally balances assignment to B across A’s arms, and vice versa. That is why simply running two tests on the same day does not, by itself, invalidate their main-effect estimates. An interaction is a difference in effects, not the mere presence of users in both tests. It matters when the conditional result changes the decision: for example, if a checkout treatment is attractive with the existing product page but not with a product-page variant the team plans to ship.

There is evidence against treating every overlap as a likely collision, though not a guarantee. Microsoft’s August 2023 analysis examined same-day test pairs across four products that independently assigned users to concurrent experiments. It detected no interactions in three products; in the fourth, abnormally small interaction p-values appeared in 0.002% of test-pair–metric combinations. Those are findings about those products, days, metrics, and detectable effects—not an expected collision rate for your program. Microsoft also acknowledges interactions the analysis lacked power to detect.

Even when there is no meaningful statistical interaction, an overlapping test’s average effect describes the mixture of other experiences present while it ran. If the other test ends, or its treatment ships to everyone, that mixture changes. A valid estimate for one launch context need not be the effect in the next one. Record the other active tests when you interpret and archive results.

Review collisions before sharing users

A launch review should ask what each variant changes, not just which page owns it. Two treatments can be on different pages and still affect the same funnel. A product-page variant may change who reaches checkout. A checkout test analyzed only among people who reached checkout can then describe a different population under each upstream experience. Specify the assignment population and an end-to-end outcome before approving that overlap; do not assume a page boundary creates independence.

Microsoft’s pre-experiment guidance recommends powering for the randomization units directly affected and logging an equivalent eligibility or counterfactual condition in control when analyzing only affected users. That matters here: if entry into a downstream analysis depends on an upstream treatment, a clean-looking checkout metric may hide a change in who was counted. Compare outcomes for originally assigned users where that answers the commercial question, and define any narrower eligible-user analysis so eligibility can be checked in both arms.

Escalate a proposed pair when treatments change the same component, user flow, eligibility rule, funnel step, downstream commercial outcome, shared code, or tracking. Shared metrics are a reason to inspect the decision, not an automatic veto. A local conversion win can conflict with a broader acquisition or repeat-purchase outcome; choose the outcome you intend to ship against, rather than treating each test’s nearest click as independent. Our guide to first-order conversion and repeat-purchase experiments covers that broader measurement problem.

Some collisions are product failures before they are statistical ones. If both variants compete for the same UI, cannot technically coexist, create a broken combined journey, or interfere with tracking or performance, QA or isolate them before exposure. A retrospective interaction check cannot repair users’ experience. For an approved overlap, write down and QA all four combined experiences, including assignment persistence, exposure events, conversion joins, and variant-specific performance. Those implementation checks are especially important on client-rendered sites; see the React and headless A/B testing guide.

Choose overlap, a joint test, or isolation

Overlap when the four experiences can safely coexist, assignment is independent and persistent, and each decision remains useful under the expected mix of the other test. This is the default for a team with scarce eligible traffic. Keep a record of concurrent tests so the readout can state which mixture was measured.

Design a joint test when the combination itself is a shipping decision. Predeclare a 2×2 factorial analysis with a treatment-interaction term. Decide what size of interaction would change the rollout, then calculate sample needs for that contrast—not just the two main effects. A methodological analysis of 2×2 trials gives approximately four times the sample to detect an interaction as large as a main effect at the same power under its assumptions. Binary outcomes, unequal allocation, correlation, and effect scale need their own calculation. A three-arm test of control, A, and B does not observe the A-plus-B combination and cannot answer the same question.

Isolate or sequence when treatments are incompatible, the combined experience is unacceptable, or a plausible interaction makes an expensive decision too risky to infer from the available traffic. Estimate the price first: show the fixed-window detectable effect and the time needed to reach the original target using eligible users, not all visits. Then use a mutually exclusive group or layer configured before launch, and verify allocation and persistent membership. Optimizely’s mutual-exclusion guidance advises exclusion only when necessary and fixed group membership from the first experiment’s start through the last experiment’s end. Statsig Layers share an allocation pool; a running experiment cannot be moved into a layer, and targeting gates do not return unused buckets to another experiment in it. Isolation needs deliberate configuration, not a last-minute switch.

Read overlapping results in the right order

Start with trust in the data. Check assignment counts and sample ratios, exposure logging, outcome joins, and performance by variant before reading lifts or interaction p-values. Microsoft’s sample-ratio mismatch guidance describes an apparent result reversed by bot filtering and warns that eligibility or exposure-based filtering can create mismatches. If an upstream treatment changes who is exposed downstream, investigate that path before interpreting a checkout-only comparison.

Next, inspect the predeclared interaction contrast where the collision review identified a decision-relevant risk. Estimate B’s lift within A’s treatment and within A’s control, then report the difference with an uncertainty interval and the four cell counts. A nonsignificant p-value is not proof of independence: interaction checks can need substantially more sample than main-effect checks. State what size of interaction the data cannot rule out. If the planned combined rollout is expensive and that uncertainty crosses the threshold for action, run a dedicated combined-variant confirmation test rather than declaring the pair safe.

Precommit the decision before seeing results. What ships if both tests win individually but the combination underperforms? What if B helps only when A is off? What if both main effects look promising but the interaction interval is too wide? Recording those rules, the active test mixture, and what changed at readout makes the result reusable rather than just another win-rate entry; the experiment-audit guide offers a framework for that record.

A launch rule your team can use

• Default to overlap. Require a short collision review for shared components, eligibility, funnel steps, commercial outcomes, code, or tracking. Name who approves an exception.

• Write down the design. Record randomization unit, eligible audience, assignment and exposure events, primary metric, guardrails, and all four possible experiences.

• Price isolation. Calculate both fixed-window detectable effect and time to the original target effect from eligible traffic.

• Match design to decision. Use independent overlap for safe, interpretable tests; power a factorial interaction when the combination decides the rollout; isolate or sequence incompatible treatments.

• Set readout and shipping rules in advance. Validate sample ratios and instrumentation first, report interaction estimates with uncertainty where justified, and reserve confirmation for consequential unresolved combinations.

If assignment, event tracking, or measurement QA makes that rule hard to operate, those are growth engineering problems to fix before increasing experiment volume.