From 12 failed generations in every 100 to fewer than 1
AI generation fails for boring reasons: providers time out, jobs vanish, finished files never reach the user. We built a reliability layer that catches those failures, retries what can be retried, and refunds what cannot. At this volume, roughly 44,000 more generations reached the user instead of ending in failure.
- Failures per 100 generations
- 12 → under 1
- 88.2% → 99.4% delivered
- Failures avoided
- ~44,000
- at measured volume
- Generations scored
- 962,376
- every run since Jun 2025, one classifier
Out of every 100 generation attempts
Each square is one generation. The dark ones failed to deliver a file to the user: a provider timed out, a job vanished, or a finished result never made it back.
The old stack lost about 12 in every 100. With the engine it is under 1. The engine does not make the models themselves better. It changes what happens when a provider fails, stalls, or loses a result.
Growth exposed how fragile the old stack was
Generation is a chain of things that can each go wrong independently: a provider queue backs up, a webhook never arrives, a file finishes but the app is never told. To the user, every one of those looks the same: nothing happened. The worst month on the old stack was February 2026, at 83.5% delivered, right in the middle of a growth spike.
Providers fail in different ways
Timeouts, refusals, malformed responses, and silent hangs each need a different recovery strategy. The old stack treated most of them the same way: stop and refund.
Completed work can disappear
Providers work asynchronously. If the completion signal is lost, the file may exist even though the app still thinks the job is running.
There was no shared definition of failure
Video, image, workflow, and voice recorded outcomes differently, so there was no honest baseline to argue from.
What the reliability engine actually does
It sits between the app and the providers, and it has one job: make sure every generation ends in a known state.
Detects
A clock notices, not a user. A sweep runs every sixty seconds and gives each attempt two deadlines: one to stop waiting and go ask the provider what is happening, one absolute ceiling for that attempt.
Classifies
Sorts the failure into one of eight categories from what the provider actually said. Two of them, a content rejection and an invalid input, mean no other provider will do better, so the job stops there and the user is told which one it was.
Recovers
Retries inside the same provider on that provider's own schedule, then moves to the next provider for the same model with no wait, since the thing that failed has already been replaced. Before giving up it asks the provider whether the job actually finished.
Resolves
Hands the main application one signed message carrying the final category, and never touches credits itself. If a result arrives after I gave up, the file is still delivered and the refund is left in place.
Four mechanisms decide what happens in the seconds after a provider fails
Each one is playable below: the rule, its real values, and the incident that forced it.
Instead of posting a guard at the door to remember who has been causing trouble, I read the last sixty seconds of the visitor log every time someone knocks.
Three failures for a model on one provider inside sixty seconds, and that pairing is skipped. Nothing stores the verdict: every check recounts the attempt log, so there is no timer to reset and nothing to lose on restart. Bad prompts never count toward it, and a single-provider model skips the check entirely.
3 fails / 60s · no state store
rolling 60s window · fails 0/3
Next in the tier list. Nothing to warm up, nothing to reset.
A rebooking desk with a printed list of partner airlines: you only ever move down the list, and the desk checks the next airline actually flies your route before booking you.
Each model carries an ordered provider list and the walk only moves forward: retries inside a provider wait five, ten, then fifteen seconds; hopping to the next provider waits zero. A prompt-caused failure stops the chain immediately, and the whole walk is capped at twenty five minutes.
up to 3 providers · 0s between hops
one job · three tiers
tier 1
trying…
tier 2
standing by
tier 3
standing by
Attempt 1 at tier 1…
Before cancelling a late delivery, the app phones the restaurant. Often the food is already on the counter, and cancelling means paying for the same meal twice.
Each attempt carries two clocks: one to go ask the provider what is happening, one absolute ceiling. The rule came from an audit, and late results are still delivered for up to forty eight hours with the refund left in place.
165 of 226 "timeouts" had finished
A returns counter that reads the reason written on the box: most phrases mean this can never be fixed, and only a few mean try again later.
Eight categories decide whether the chain continues, what the user is told, and whether that provider's health takes the hit. The first version matched one word; it is now 127 phrases, each pinned to the production failure that forced it.
8 categories · 127 stop-patterns
what the provider said → what the engine does
8 categories · 127 phrases · each pinned to the production failure that forced it
Carved out of the monolith, and it keeps no memory of its own
The engine is its own Cloudflare Worker. It owns provider choice, the attempt log, and the operator API; money, accounts and policy stayed in the main application. Flip the toggle to see what moved.
Mechanics, not policy
Billing, credits and refunds stayed behind, so the engine can be redeployed mid-incident.
Stateless
No cache, no KV: state is Postgres. The only memory is a five second health note.
Never touches money
One signed callback with five outcomes; the refund decision lives on the other side.
Failover is data
Provider order, timeouts and retries are rows edited in the console. No deploy.
I built the screen I wanted at three in the morning
A working miniature of the console's reliability tape. One row per model, always sixty cells wide whatever the window, each cell coloured by the worst thing in it.
reliability tape · one row per model
60 × 12-hour buckets, same width either waydark cell = the worst thing in that bucket<95% warns · <80% pages · min 20 outcomes
What I left out of the headline number, and why
Every one of these is a switch in the explorer below, so you can put them back and watch the number move.
- Moderation blocks are excluded
- If a prompt is refused for content, the system did its job. Counting that as a failure would make safety look like breakage. Including them lowers both eras, and the gap barely moves.
- Onboarding demos are not counted at all
- They replay a fixed template instead of running a model, so they are not generations. They are out of the dataset entirely, not just filtered off.
- Deleted videos still count as successes
- If a generation produced a file, it worked. Deleting your own video afterwards is a preference, not a system failure.
What is still not good enough
The remaining failures are not spread evenly. They cluster in a small number of legacy providers and older model configurations that have not been migrated yet. That is the next piece of work, and the explorer below will show it the moment it improves, or does not.
One inconsistency I have not fixed yet. The console subtracts both user-caused categories from the success rate it shows, while the two alerting jobs subtract only one. A model swamped by bad input can therefore read healthy on screen and still page someone. It is written down here because a page that asks you to argue with its numbers should show the seams too.
A note on the numbers: the headline compares the eras before and after launch, and skips May 2026 because the engine launched partway through that month. The large figure below summarises the full selected range, transition included, so the two will differ.
93.5%
962,376 generations in this view · 5 Jun 2025 → 31 Aug 2026
88.2%
Before the engine
Jun 2025 – Apr 2026
99.4%
With the engine
Jun 2026 – today
Loading the interactive report: filters, chart views, and the full breakdown appear here.
Share of generations that delivered a file, by month, over all time. It started at 90.0% in Jun 2025 and sits at 99.4% in Aug 2026, a move of +9.4 points.