A 202 Response Does Not Make Your Background Job Durable

A 202 Accepted response can leave a client holding a job ID for work that no longer exists. If the work matters, persist its intent before acknowledging the request, then design for retries, duplicate execution and a completion state the caller can verify. Moving work out of the request handler is not the same as making it durable.

An accepted request is not a persisted job

Consider an endpoint that receives POST /exports, schedules a background task and returns 202. The task begins generating a customer export inside the API process. A rollout terminates that process before the file is ready. The replacement Pod serves new requests, but it has no record of the export to resume. The client received acceptance; the system retained no recoverable obligation.

That is consistent with the tools’ contracts. Starlette’s BackgroundTask runs in-process after the response is sent, and FastAPI’s implementation uses it. Neither creates a persistent job record. Kubernetes Pods are ephemeral: a replacement does not inherit the terminated process’s memory. Even an orderly termination does not solve the general case. Kubernetes normally allows a 30-second termination grace period, shared by a PreStop hook and shutdown. It may let some work finish, but it cannot restore work after a node failure or guarantee completion when shutdown takes longer.

Nor does the response code fill the gap. HTTP 202 says processing has been accepted, not completed; it says nothing about where the work is stored. Treat acceptance, durable scheduling, execution, external side effects and verified completion as separate events. If clients depend on eventual completion, the API should acknowledge only after it has durably recorded what it has agreed to do.

Write the survival contract before choosing a tool

Start with a product question: what is the consequence if this task disappears? Refreshing a disposable cache may be explicitly best-effort. Sending an invoice, provisioning an account or generating a paid-for export probably is not. A task that can fail and wait for a person to resolve bad input needs a different end state from one that can be retried until it succeeds.

For consequential work, define an operation record with a stable ID, request parameters or a reference to them, creation time and status. Persist it before returning 202; return the ID and a status location. A useful status contract distinguishes queued, running, succeeded, failed and intervention_required. Define what each means to the caller. In particular, succeeded should mean the intended business result is durable—not merely that a worker started or an HTTP request was sent.

Decide whether callers may retry creation. If a client times out after the server commits the operation but before receiving 202, it may send the same request again. A client-supplied idempotency key, scoped and enforced by your API, can return the existing operation instead of creating another. Without that contract, clients cannot safely distinguish a lost response from a rejected request.

Then assign ownership: which component retries a queued operation, which errors are permanent, when does work require intervention, and how does an operator find it? Do not promise “eventual completion” if the real policy is three attempts followed by a dead-letter queue nobody reviews. An explicit failure state is more honest—and more useful—than an operation that remains running forever.

Choose the smallest execution model that meets the contract

FastAPI describes BackgroundTasks as a simple option for small tasks and points to a separate task system for heavier work across processes or servers. That is a sensible boundary, not a rule that every email or cache refresh needs a workflow engine. If losing a best-effort notification is an accepted product decision, document it and keep the simpler implementation.

For recoverable discrete work, a database operation record plus workers often gives the clearest contract. A queue can distribute that work and absorb bursts. For example, an SQS standard queue stores messages across multiple availability zones before acknowledging a send. But the API must wait for that successful send before claiming the message is queued, and SQS can deliver duplicates or messages out of order. A durable operation record remains valuable for client-visible status and business-level deduplication.

Watch the boundary between your database and queue. If the API commits an order and then sends a message, a crash between those actions leaves an order with no queued work. Reversing the order creates its own inconsistency. A transactional outbox records publication intent with the business change, then publishes it separately. It closes the database-commit-versus-send gap, but consumers still need to handle repeat publication; Debezium’s outbox router can include an event ID for deduplication.

A Kubernetes Job is reasonable for bounded batch work when you already operate Kubernetes. It tracks completions and starts another Pod when one fails. It does not replace your operation record, status API or coordination across several external systems. Kubernetes explicitly warns that even a single-completion Job may sometimes start the program twice.

Use a durable workflow when the operation’s state machine—not just its runtime—is the hard part: multiple dependent calls, a wait for approval, or recovery after days without holding a worker. Temporal retries failed Activities by default, while Workflow Executions have no retry policy by default. That distinction is a reminder to place failure-prone calls in the right unit and make their implementation idempotent. A workflow runtime can remember progress; it cannot make a payment provider forget a charge.

Design for the retry after the side effect

The dangerous sequence is straightforward. A worker claims a message, calls a supplier, and the supplier performs the action. The worker crashes before recording the result or acknowledging the message. The message becomes available again, another worker calls the supplier, and the action happens twice. With SQS visibility timeouts, an undeleted message can be delivered again when its timeout expires; the default is 30 seconds and can be adjusted or extended.

Use the operation ID to derive a stable idempotency key for each business action, and pass that same key on every retry. The receiving system must enforce the key. For your own database writes, enforce the invariant with a durable uniqueness constraint or equivalent transaction design. Record the outcome before acknowledging queue work, so a retry can recognize completion. If the external service cannot deduplicate, you need a way to query or reconcile its state before repeating an action whose result is unknown.

A lost response is enough to create that unknown state even without a worker crash. The external service may have succeeded while the network made it look like a timeout. Durable execution replay can restore recorded step results after a crash, but a side effect whose response was never recorded may run again. The same limit applies to Lambda durable functions: deduplicating execution starts does not change the default at-least-once semantics of steps. Do not sell “durable” as “exactly once.”

Set operational limits alongside idempotency. Choose a visibility timeout or lease appropriate for the work, renew it for long tasks, and cap worker concurrency to protect downstream services. Classify retryable errors separately from invalid input or a permanent rejection. Set attempt limits and a reviewed dead-letter or intervention path. Otherwise a poison message can cycle indefinitely, or a transient outage can quietly become a permanent business failure.

Test the contract by interrupting it

A status endpoint should expose the operation’s actual business state, including a terminal failure or intervention state. Monitor queue age, repeated failures and operations stuck in running; an empty queue alone does not prove completion. Where execution telemetry is available, keep it distinct from business status. AWS’s Lambda durable-functions telemetry guidance distinguishes execution status from an individual invocation, and notes that its exporter cannot observe terminal states that occur without another invocation. Logs and traces help investigate a job; they are not its completion record.

Before relying on the design, run these interruption tests:

• Kill the API process immediately after it returns 202. Can another process find and complete the operation?

• Retry the creation request after losing its response. Does the same idempotency key return the same operation?

• Terminate a worker before and after each significant external call, then before acknowledgement. Does recovery avoid duplicate business effects?

• Deliver the same message twice and let a lease expire mid-task. Do concurrent workers preserve the same invariants?

• Drop an external API response after the receiver acts. Can the worker reconcile an unknown outcome rather than blindly repeat it?

• Force permanent failure and exhaust retries. Does the status change, does an owner receive an alert, and is there a safe intervention procedure?

If those tests expose gaps between the API contract and the worker implementation, Makers’ Den’s backend development team works on the APIs, queues, retries, idempotency, dead-letter handling and observability needed to close them.