Retry Budgets: Fit Backoff Inside the User Deadline
Set retry budgets that respect deadlines, idempotency, and downstream load. Work through nested retries, ambiguous failures, and a practical decision table.
A retry budget is the amount of additional work and time you permit after an operation fails. Exponential backoff is only one part of that budget. Without a total deadline, a polite-looking retry loop can keep working long after the user has stopped waiting.
Imagine a checkout page with a two-second budget for a nonessential shipping estimate. I would decide first whether a missing estimate should block checkout. That product decision determines the fallback, which in turn determines how much time a retry is worth.
Classify the failure before retrying it
Microsoft's Retry pattern guidance distinguishes transient failures from lasting failures, discusses idempotency, and warns that retries at multiple layers can introduce excessive delay. Google Cloud Storage's retry strategy documentation similarly separates retryable conditions from operation idempotency and describes bounded backoff policies.
Those sources describe their contexts. A status code alone should not overrule the contract of your own dependency. In my proposed shipping integration, the decision table would be:
| Result | Proposed action |
|---|---|
| Invalid destination input | Return validation feedback |
| Authentication failure | Surface an operational error; do not loop |
| Temporary service unavailable | Retry only within the remaining budget |
| Caller cancellation | Stop work |
| Unknown outcome from a state-changing call | Reconcile or use an idempotency mechanism |
The table is intentionally specific to the operation. “Retry every exception three times” erases the distinction between a recoverable network interruption and a request that can never succeed.
Spend one shared deadline
For the hypothetical two-second operation, reserve time for the application to render a fallback. Suppose the first attempt consumes 700 milliseconds and a chosen delay consumes 300. Only one second remains before the original deadline, less the fallback reserve.
Starting a new two-second timeout for the second attempt would silently extend the user-visible operation. Instead, compute each attempt's allowance from a monotonic deadline established before the first attempt.
Illustrative pseudocode:
deadline = monotonic_now() + total_budget
attempt = 0
while attempt < maximum_attempts:
remaining = deadline - monotonic_now()
if remaining <= fallback_reserve: stop
result = call_with_timeout(min(attempt_cap, remaining - fallback_reserve))
if success: return result
if not safe_and_retryable(result): stop
remaining = deadline - monotonic_now()
delay = choose_bounded_jitter(attempt)
if delay + minimum_useful_attempt >= remaining - fallback_reserve: stop
wait_with_cancellation(delay)
attempt += 1
return fallback
This is a design sketch, not executable code. A real implementation must propagate cancellation through both waiting and the underlying request, and ensure the transport actually enforces its timeout.
Add randomness without losing the bound
For this proposed policy, choose a random delay between zero and a capped exponentially increasing ceiling. The purpose is to avoid scheduling every caller's next attempt at the same instant after a shared failure.
The randomness must remain inside the total deadline. A large maximum backoff is irrelevant if the interactive operation has only a short time remaining. Conversely, a background reconciliation job can use a longer policy because its consumer is not a person staring at a spinner.
I would keep interactive and background retry configurations separate, even when both call the same service. Their useful completion windows differ. Sharing one helper should not force them to share one latency contract.
Count retries across layers
Suppose an application makes up to three attempts, an SDK makes up to three attempts for each, and a proxy also retries each failed request up to three times. In the worst-case shape, one logical operation can generate 27 underlying attempts. This is illustrative multiplication, not a claim about any named SDK's defaults.
Inventory retries in the browser, application, SDK, proxy, worker, and queue. Assign a clear owner for the policy where possible. If a lower layer must retry, include that behavior in the upstream deadline and telemetry.
A useful metric is attempts per logical operation. Pair it with final success rate and end-to-end latency. A higher success rate can conceal an unacceptable increase in downstream load or user wait time.
Treat ambiguous writes differently
A timeout does not tell you whether a remote service applied a write before the connection failed. For a shipping quote this may be tolerable; for purchasing postage it is not. My proposed design would use a stable business operation ID and the provider's documented idempotency support, or query the operation's status before attempting another purchase.
Do not generate a new idempotency key for each retry of the same intended action. Also define when a genuinely new action requires a new identity. Those rules belong beside the endpoint contract, not buried inside a generic retry library.
Use the webhook idempotency design to reason about ambiguous completed writes. Coordinate retries with the rate-limiting policy so clients do not amplify an already overloaded dependency.
I would ship retries only after a failure-injection exercise demonstrates bounded attempts, bounded elapsed time, and the intended final business state. Success after an unlimited wait is not the same as a reliable user experience.