Most AI products look dependable in a demo because the model answers quickly, the tool calls succeed, and only one person is using the system.
Production removes all three assumptions.
A model provider slows down. A rate limit arrives during a traffic spike. A tool succeeds, but the response disappears before the application receives it. Ten users ask for expensive work at once. The model returns an answer, but not in the format the next step expects.
The visible feature may be AI. The thing keeping it useful is ordinary reliability engineering.
A reliable AI product is not one whose model never fails. It is one that knows how to fail deliberately.
The model is a dependency, not the product
When teams wire a model call directly to a button, they make the entire product inherit the behaviour of a remote dependency: its latency, quotas, incidents, and output variance become the user experience.
That is acceptable for a prototype. It is a fragile production boundary.
The application should own the request before the model sees it. It should assign an identity to the work, validate the input, decide whether the task is synchronous or asynchronous, enforce a time budget, and record enough state to recover. The model is then one worker inside the system—not the system itself.
This separation changes the failure question from “Is the AI up?” to “What useful state can the product preserve when this dependency is impaired?”
Decide what must be immediate
Not every AI interaction belongs behind a loading spinner.
Autocomplete, classification used during checkout, and a short conversational reply may need a synchronous answer. A research report, document extraction, media generation, or multi-step agent run usually does not. Forcing long work into a single HTTP request creates a brittle chain: browser, application server, model API, tools, database, and response stream all have to remain healthy at the same time.
For work that can finish later, accept the request, persist it, place it on a queue, and return a job state immediately. A worker can process it when capacity is available while the interface shows honest progress.
Queues are especially useful when arrival rate and processing rate are different. AWS documents this pattern for absorbing load spikes and allowing buffered requests to be handled independently. But a queue is not permission to build an infinite backlog. Track the age of the oldest job, expire work that is no longer valuable, and send repeatedly failing jobs to a dead-letter path for inspection.
The product decision is simple:
- If the user needs the answer now, keep the path short and tightly bounded.
- If the work is expensive or multi-step, make it resumable and observable.
- If an old result is useless, give the job a deadline instead of processing it eventually.
Put a time budget around every dependency
A slow answer can be more damaging than a fast error. It occupies connections, consumes worker capacity, and encourages users to retry—creating more work while the system is already unhealthy.
Every remote call needs a timeout derived from the experience you can still deliver, not from the maximum patience of the underlying SDK. A ten-second product budget cannot contain three dependencies that are each allowed to wait ten seconds.
Retries also need a budget. Some failures are transient, so retrying can recover them. But immediate, synchronized retries can amplify an incident into a retry storm. The safer pattern is a small number of attempts, exponential backoff, and jitter so clients do not all return at the same moment. This is a core pattern in the AWS Builders’ Library guidance on timeouts and retries.
Retry only when the error is plausibly temporary. Invalid input, exhausted context, a permission failure, or a request that violates policy will not improve because the same payload was sent again.
Make repeated work safe
A timeout creates ambiguity. The client knows it did not receive a response; it does not know whether the operation completed.
That distinction matters when an AI workflow can send an email, update a CRM record, place an order, or create a billable job. A blind retry could perform the action twice.
Give each operation an idempotency key and store the result against it. If the same request arrives again, the system can return the earlier result or continue the existing job instead of creating another one. AWS describes the same principle in its guidance on making retries safe with idempotent APIs.
Idempotency is not only a payments concern. It is what lets AI workflows recover from uncertain network outcomes without duplicating real-world consequences.
Stop sending traffic to a dependency that is already failing
If a model endpoint is timing out repeatedly, continuing to send it full traffic wastes the remaining capacity and lengthens every request behind it.
A circuit breaker turns repeated failures into a temporary routing decision. After a threshold is crossed, the application stops calling the impaired path for a short period. Requests can fail fast, enter a queue, use a compatible alternative, or receive a reduced experience. A controlled probe later determines whether the dependency has recovered.
The important detail is scope. Breakers should isolate a model, region, feature, or tenant where possible. One degraded capability should not take down unrelated work.
This is the same reason mature systems use bulkheads: capacity for one workload is kept separate from another. A flood of document-generation jobs should not consume the workers needed for authentication, billing, or basic account access.
Design the degraded experience before the incident
“Try again later” is sometimes correct. It should not be the only fallback a product knows.
Useful degradation depends on the feature:
- A support assistant can switch from generated synthesis to verified help-centre results.
- A document workflow can save the upload, mark analysis as delayed, and notify the user when it finishes.
- A recommendation feature can serve a recent cached set instead of an empty page.
- An agent can return the completed steps, unresolved items, and a resumable job rather than discarding the entire run.
- A high-stakes action can prepare a draft and move to human review instead of attempting uncertain automation.
Fallbacks should be simpler than the primary path. A second system with the same dependencies, latency profile, and failure modes is not a fallback; it is another branch of the incident.
Multi-model routing can help, but it adds its own contract problem. Providers differ in tool calling, context limits, structured output, safety behaviour, and latency. A fallback model is only real if it is exercised continuously with the same evaluations as the primary one.
Measure whether the user received value
Infrastructure dashboards can remain green while an AI feature is failing.
The API returned 200, but the output did not match the schema. The agent completed, but repeated a tool call twelve times. The fallback answered, but quality dropped below the point where a user could act on it.
Alongside uptime and latency, track product-level signals:
- time to first useful output, not only total response time,
- completion rate by model, feature, and failure class,
- queue depth and age of the oldest job,
- retry volume and duplicate suppression,
- structured-output validation failures,
- fallback activation and fallback success rate,
- cost per successful outcome,
- runs resumed successfully after interruption.
Trace the request across the application, model call, tool calls, and final action using one correlation ID. Without that thread, teams see five partial logs and still cannot explain what happened to the user.
A production reliability checklist
Before an AI feature carries real workflow responsibility, it should answer these questions:
- What is the maximum useful time for this request?
- Which work must be synchronous, and which work can be queued?
- What errors are retryable, and how many attempts are allowed?
- Can the operation be repeated without duplicating side effects?
- What happens when the primary model is slow, unavailable, or malformed?
- Which parts of the product remain usable during that failure?
- Can an interrupted job resume from a checkpoint?
- Will the user see an honest state and a next action?
- Can the team trace one failed request end to end?
- Is success measured as a useful outcome rather than a successful API call?
Reliability lives around the model
Models will improve. Providers will add capacity. None of that removes networks, quotas, traffic spikes, partial failures, or unpredictable work.
The durable advantage is the layer around the model: clear time budgets, bounded retries, safe repetition, workload isolation, resumable state, and a degraded experience that still respects the user’s time.
That layer is less visible than the prompt. It is also the difference between an AI demo and a product people can depend on.