A Conversion Test Can Win the First Order and Lose the Customer

A first-order conversion lift is not enough to approve an acquisition offer in a repeat-purchase business. The offer may change who buys, what they spend and whether they come back. Decide what would count as a commercial win before assigning traffic, then keep the original experiment groups intact long enough to measure it.

The first checkout is not the whole outcome

In a supplement ecommerce test, a shorter subscription-plan option produced more transactions but a lower average order value. Retention was not established; the account’s estimate of future customer value was a forecast. Speero’s account largely reports first-purchase evidence, not completed reorder experiments. It does not show an offer that won on first orders and later lost on observed repeat purchasing.

That distinction matters. The result establishes a measurement problem, not a cautionary tale with a known ending. If a different plan brings in buyers who place fewer or less profitable later orders, a conversion win could become a commercial loss. If those buyers reorder profitably, the lower first-order value might be acceptable. Neither conclusion follows from the initial transaction count.

Even earlier funnel gains can mislead. In another test Speero describes, a homepage change increased add-to-cart events and checkout starts while completed purchases fell. That is evidence that funnel stages can move in opposite directions; it is not evidence about later orders. Treat each stage as a diagnostic for a different question, rather than allowing the most encouraging chart to define success.

The right primary outcome depends on the intervention. A checkout usability fix can reasonably target completed first purchases, with guardrails for refunds and commercial performance. An acquisition offer, price or subscription-plan change alters the economics of the customer relationship itself. For that decision, first-order conversion is an early signal; observed contribution across the assigned audience is the stronger commercial test.

Define three outcomes before assigning traffic

Write the scorecard before launch. Start by defining eligible visitors: are they people who have never bought, existing customers considering another order, or a mixture? If the question is whether an acquisition offer brings in valuable new customers, mixing existing buyers into the primary analysis makes the answer harder to interpret. Record eligibility, the assignment unit, what counts as exposure and when the observation window begins.

For an acquisition test, use three separate layers:

• First-order completion: the share of eligible assigned visitors who complete an initial purchase. Report completed purchases, not just checkout starts, and retain visitors who buy nothing in the denominator.

• Observed second order: whether an assigned visitor places a second purchase within a predeclared window. Report this across everyone originally assigned. Also show reorder rate among first-time buyers, clearly labeled as a diagnostic.

• Observed contribution: contribution from orders attributed to each originally assigned visitor over that same window, including zero for nonbuyers. Define the included revenue, discounts, refunds and costs before seeing results.

Why report second orders twice? A reorder rate among buyers answers whether the customers who purchased came back. It does not answer whether the offer created more valuable customers per eligible visitor. The variant can change who becomes a first-time buyer. Comparing only those buyers conditions on an outcome the experiment itself affected; it can make two groups that began comparable look meaningfully different for reasons other than subsequent retention. Speero recommends following everyone originally assigned, including nonbuyers, for an acquisition-offer experiment. If the question is instead whether onboarding improves retention, randomize eligible purchasers separately.

Contribution needs an explicit definition. Product cost, payment fees, fulfillment, shipping subsidies and acquisition spend may all matter, but they will not all live in the same system or be attributable with the same confidence. State which are included and where coverage is incomplete. ShopifyQL’s sales schema exposes first-time and returning orders, discounts, cost of goods sold and gross profit. Shopify defines gross profit as net sales minus cost of goods sold, before other operating expenses. It is useful source data, not automatically contribution after fulfillment or acquisition costs.

Choose one primary decision metric for the offer test and a smallest worthwhile effect before launch. Specify rollback guardrails: for example, which deterioration in completed purchases, refunds or contribution would make an apparent win unacceptable. Plan for enough affected visitors to evaluate the chosen effect, and show uncertainty alongside the estimate. Microsoft’s pre-experiment guidance calls for a falsifiable hypothesis, success and guardrail metrics, a power calculation and a suitable randomization unit. A favorable direction on a thin sample is not a decision rule.

Keep the experiment attached to the customer and the order

The analysis depends on a join that survives checkout and later purchases. At minimum, retain the experiment ID, assigned variant, assignment identifier and exposure record. Where permitted, connect that record to a stable customer identifier and backend orders, including order ID, order time, line items, discounts, refunds and the cost fields needed for the agreed contribution definition. Keep experiment IDs and order IDs distinct: one identifies an assignment, the other a transaction.

Persist assignment in an appropriate first-party system rather than expecting an analytics cookie to reconstruct it months later. A visitor may clear cookies, move between devices, check out as a guest or create an account after the first purchase. Those are join and coverage problems to measure, not edge cases to wave away. Microsoft Research identifies unstable cookies, survivorship bias and selection bias as pitfalls in long-term experiment analysis.

GA4 can help diagnose the journey. Its ecommerce events distinguish add_to_cart, begin_checkout, purchase and refund; its purchase example includes transaction_id, value, currency, coupon, items and customer type. Those events do not supply your contribution calculation. Reconcile tracked purchases and refunds against backend orders by variant, rather than treating the analytics revenue report as the financial ledger.

Pay particular attention to transaction_id. Google says matching IDs deduplicate GA4 purchases and that the ID must be unique to each order; an empty ID can incorrectly collapse purchases. Check for missing IDs, duplicates and variant-specific event loss. Also check whether assignment survives the path from offer to checkout. A perfectly implemented scorecard cannot rescue an experiment whose variants were not delivered or recorded consistently.

GA4 User-ID can connect activity across sessions and devices when the business consistently supplies its own unique IDs. Google cautions against identifiers that disclose identity and notes that newly implemented User-ID does not retroactively associate previously collected data. Use only permitted identifiers, and document how much of each variant remains joinable. Guest checkout, retail and marketplace purchases may not be linkable to website assignment at all. Do not turn an incomplete join into a claim about complete customer lifetime value.

Avoid a misleading retention win

Set the follow-up window from the product’s actual replenishment cycle, not from the date the early conversion chart looks convincing. For an acquisition test, start that window at assignment. Wait until each assigned cohort has had the full opportunity to reorder before making the later-order comparison. Label newer cohorts as immature and publish their first-order results as provisional.

Predeclare how cancellations, refunds, subscription pauses and late-recorded returns affect the readout. Report join coverage and missing cost coverage by arm; if one variant sends more people through a path you cannot track, the missing data is itself part of the decision. Keep forecast lifetime value separate from observed contribution. A forecast may help plan a follow-up, but it cannot stand in for orders that have not happened.

Longer observation is not automatically more trustworthy. Identity loss accumulates, cross-device gaps remain, and not every customer’s reorder window has elapsed. A later report that silently drops unjoined visitors or keeps only purchasers can be more biased than the early read. Preserve the original assignment population, show nonbuyers as zero outcomes and make exclusions visible. For a more detailed approach to deciding what an A/B test actually taught the team, see our guide to auditing test learning rather than win rate.

A preflight checklist for the next offer test

Before traffic reaches the variant, have the growth, engineering and commercial owners agree on the following:

• Decision: State the offer hypothesis, eligible audience, primary metric, smallest worthwhile effect, observation period and rollback guardrails. Decide whether the test is about acquisition economics or a narrower checkout fix.

• Assignment: Record the experiment ID, variant, randomization unit, exposure definition and stable assignment identifier. QA allocation and confirm that exposure reflects what visitors actually saw.

• Orders: Verify joins from assignment through checkout to backend customer and order records. Reconcile GA4 purchase IDs with authoritative orders; check guest-to-account transitions, consent constraints, duplicates and variant-specific tracking loss. Our guide to A/B testing React and headless websites covers implementation concerns along that path.

• Economics: Document the contribution formula, cost sources, refunds and exclusions. Show first-order completion, second orders and contribution as distinct outcomes, with uncertainty and missing-data coverage by arm.

• Follow-up: Set the window from the replenishment cycle, retain everyone originally assigned and wait for cohort maturity. Label early results, reorder rates among buyers and lifetime-value forecasts for what they are.

If assignment, tracking or checkout-to-order joins cannot support that readout yet, Makers’ Den’s Growth Engineering service can help implement and validate the experiment and scorecard. The offer strategy, statistical sign-off and privacy decisions still belong with the client’s appropriate owners.