A test that takes a database offline before an order request proves something useful: the service handles an unavailable database. It does not prove what happens when the database commits the order, its acknowledgement arrives after the client’s deadline, and a retry overlaps another request. For critical workflows, test those event orders deliberately. Use deterministic simulation to make them repeatable, and keep real-infrastructure tests for behavior the simulation cannot model.
A clean dependency failure is the easy case
Consider an order API. The first request carries an idempotency key and reaches storage. Storage commits the order, but the acknowledgement is delayed. The client’s deadline expires, so the client retries with the same key. Meanwhile, another order request is in flight. The delayed acknowledgement then arrives, possibly after the retry has begun or completed.
What may the client observe? It might receive a timeout for the first attempt and a successful response for the retry. It might receive a late response that its client library discards. If the service cannot yet determine the operation’s outcome, it may return an explicitly ambiguous result and require the client to query by key. Those outcomes need a documented contract. What must not happen, if the API promises idempotent order creation, is two durable orders for the same key. The concurrent, unrelated request must not be mistaken for the retried order either.
A conventional integration test often makes the dependency fail before sending the request, then checks for an error response. That exercises neither a successful commit followed by a lost acknowledgement nor the race between a retry and the original attempt’s completion. Even injecting the same fault with the same random seed does not necessarily reproduce the race: live threads, network delivery and disk completion can change which event wins. Unmesh Joshi’s account of deterministic simulation draws precisely that distinction between controlling faults and controlling execution.
This is also why HTTP-status assertions are too weak. A green response can conceal a duplicate side effect; a timeout can follow a successful commit. Our discussion of HTTP wrappers, 503s and deadlines is a useful reminder that changing client error behavior can change retry decisions. To establish what the service guarantees, inspect its client-visible history and durable effects, not only the response from one attempt.
Specify the contract before scheduling faults
The test needs a property to check, not a vague expectation that the service ‘recovers.’ For the order example, start with: every request using the same valid idempotency key refers to at most one durable order creation. Define when the key becomes reserved, when the operation becomes committed, what a retry returns while an earlier attempt is still running, and when an ambiguous result is permitted. Check that a separate key can still create its own order.
Those choices shape the implementation. AWS’s idempotency guidance recommends recording the token and operation state with concurrency control so simultaneous arrivals cannot both perform the mutation. A test should therefore inspect the persisted operation state as well as the API result. If order creation triggers another side effect, the property must include that effect too; deduplicating the API response alone is not enough.
Make the retry policy part of the contract. Which layer retries: the client, API gateway or service? What is the attempt budget? Can a deadline expire while an attempt is still running? AWS warns that retries at multiple layers compound and that retrying non-idempotent operations can duplicate effects. Backoff and jitter help manage load, but they do not make an unsafe mutation safe. Decide where retries belong, bound them, and test the path where an attempt may already have committed.
Avoid asserting a stronger consistency model than the product promises. If the service promises a linearizable read after an acknowledged write, a stale read is a defect. If it explicitly promises weaker consistency, that same observation may be allowed. Write the permitted histories down before building a checker; otherwise a simulation can confidently report the wrong result.
Make the dangerous interleavings replayable
The useful simulation boundary is usually smaller than the whole backend. Extract the state transitions for one workflow behind interfaces for time, dependency calls and persistence. Let a test control when a write starts, when it becomes durable, when its completion is delivered, when a deadline fires and when a retry becomes runnable. Production code should still use the real clock and adapters; the test supplies controlled ones.
Begin with a named schedule for the order case: commit the initial write, withhold its acknowledgement, advance virtual time past the client deadline, start the retry, allow a separate request to progress, and then release the old acknowledgement. Repeat with the separate request and acknowledgement in different orders. Record each invocation, completion and durable transition so the failure can be explained, not merely reproduced.

Then add seeded exploration around those controlled completion points. Vary message delivery, timeout, retry initiation and recovery while checking the same domain property after every run. Keep each discovered failure as a named regression schedule. TigerBeetle’s VOPR documentation describes a more extensive version of this approach: substituted clock, network and disk behavior; seed-based fault variation; replay using the seed and Git commit; and checks for safety and liveness rather than response counts alone. A service team can adopt the principle without adopting that scope.
The hard requirement is control of relevant execution, not just fault injection. If code still reads wall-clock time, launches uncontrolled threads, draws unseeded randomness or depends on unordered iteration, the same seed may not replay the failure. FoundationDB’s client-testing documentation explicitly warns that saved seeds only reproduce simulations when workloads avoid nondeterministic behavior, and treats multithreaded races as a separate real-cluster testing concern. Save the code revision, scenario configuration and execution history alongside the seed. A seed without its code and inputs is a poor incident report.
Do not assume this is a test-runner switch. Tickloom’s README presents a Java simulated network, cluster testkit and seeded runner, with Java 21+ and an example 0.1.0-alpha.7 dependency. It is a useful teaching reference, not evidence that an established multithreaded service becomes deterministic by adding a dependency. Retrofitting every scheduler, driver and clock read may cost more than the workflow warrants. A bounded state machine with controlled adapters is the better pilot.
Keep the real system in the test plan
Simulation can establish that a property holds for the schedules and component behavior its model actually explores. It cannot prove that all schedules were explored, that a database driver handles connection loss as modeled, or that storage and sockets behave exactly like their simulated counterparts. A passing run is evidence about a model, not a certificate for production.
Keep a separate integration lane against the actual database, broker and network stack. Check transaction boundaries, serialization, driver retries, connection handling and restart recovery. Use live fault tests for particularly consequential behavior when their runtime and operational risk are justified. If production uses multithreaded clients, exercise those too rather than assuming a single-threaded simulator covers their races.
This separation has precedent. FoundationDB documents a single-threaded deterministic cluster simulation using the same Flow asynchronous runtime as production, while also using live performance and hardware-based failure tests. Sharing runtime code makes that simulation unusually valuable; it still does not replace real systems.
Real-cluster investigation can expose interactions a tidy model misses. In a historical Jepsen investigation, RethinkDB 2.2.3 had passed earlier partition tests but lost acknowledged operations when partitions overlapped cluster reconfiguration. Jepsen reported that the triggering behavior was difficult to reproduce; fixes followed in 2.2.4 and 2.1.6. This is not a claim about current RethinkDB. It illustrates why testing one failure at a time can miss a damaging overlap. Jepsen’s real-cluster method generates concurrent client histories and checks them against a model; it can find errors, not prove their absence. Deterministic simulation and real-cluster testing answer complementary questions.
Production-traffic replay is another useful but distinct lane. The AWS Architecture Blog’s replay framework compares captured responses with responses from changed code. That can catch output regressions, especially for stateless, computation-heavy services. It does not, by itself, arrange a lost acknowledgement alongside a timeout and concurrent retry, and stateful requests need safeguards against repeated side effects. Calling both approaches ‘simulation’ should not collapse their different purposes.
Start with one reliability decision
Do not begin by commissioning a general-purpose simulator. Begin with the failure whose cost would justify making its execution reproducible. If duplicate charges are the concern, use that workflow rather than a convenient read-only endpoint. If an existing service has uncontrolled concurrency throughout its core, first isolate the decision and state transitions that govern the charge. Leave broad scheduler control for a later decision, if the pilot earns it.
Assign ownership of both test lanes. The team changing retry policy must update the simulated schedules and the real-client tests; the team changing database or broker behavior must revisit the adapters and integration cases. If you need help implementing those seams and operating the layered test strategy in an existing service, that belongs in backend development, alongside idempotency, retry handling and production observability—not in a test-only project detached from the service contract.
Use this pilot checklist:
Choose one invariant: State the durable effect and the client-visible outcomes permitted after a timeout, including any ambiguous outcome.
Map the workflow: List operation states, idempotency keys, dependency completions, deadlines and the retry layer and budget.
Control the schedule: Run a named lost-acknowledgement case with a retry and another request; vary their completion order under a seed.
Keep the evidence: Save the seed, scenario parameters, execution history and code revision. Turn every discovered failure into a fixed regression schedule.
Verify outside the model: Exercise actual drivers, transactions, connections and recovery. Review which production behavior the simulation does not represent before treating a green run as a release signal.