The typical AI workflow now finishes 40 seconds sooner
A workflow run is a relay: five AI steps handing work to each other. Most of the waiting was not the AI thinking. It was the gaps between the handoffs, where work sat in a queue or took the long way across the network. I fixed the gaps and left the AI completely alone.
- Typical run
- 40s sooner
- 340s → 300s · median
- Slowest 10% of runs
- 87s sooner
- 655s → 568s · p90
- A lost result used to surface in
- up to 14 min
- now recovered in ~66s
The same run, before and after
Watch both runs go. The solid blocks are the AI generating, identical in each track, because I never touched them. The hatched slivers are waiting: time spent in a queue, crossing the network, or writing bookkeeping. The bottom track finishes earlier only because those slivers shrank.
Two simulated workflow runs on a shared time axis: before (340s) and after (300s). The grey segments are AI generation, identical in both. The hatched segments are pipeline overhead.
Forty seconds compounds quickly. For someone producing a dozen workflows in one session, it removes several minutes of dead waiting. The runs that were already slowest improved the most, because that is where the waiting had piled up.
Four places the time was going
None of these were the AI being slow. Each one showed up in the traces before anyone had a theory about it. Open any row for the detail.
A bus that waits five seconds at every stop, even when nobody is boarding.
Work moved between steps through a queue set to wait up to five seconds before handing anything over, even when a message was already sitting there ready. A five-step run paid that toll five times, for nothing.
I set the wait to zero and split the one shared queue into three lanes: long provider jobs, short "it's done" notices, and status checks. A two-second notification can no longer sit behind a ninety-second video job.
up to 5s × 5 steps
Two people in the same building, mailing each other letters.
All three services run side by side, but they spoke to each other over public internet addresses: out of the building, around the block, back in. Sometimes a message got lost out there. Worse, every "your video is ready" notice was relayed through a third service acting as middleman.
They now talk over internal bindings: direct calls that never leave the network. The middleman was measured, tested on 10% of traffic first, then deleted once it had proven unused for three days.
3.5s → 1.6s per handoff
Your video was finished. Nobody had told the app yet.
When a completion notice went missing, a cleanup job ran every five minutes and was slow to act on what it found. A finished video could sit invisible to its owner for seven to fourteen minutes.
A reconciler now runs every sixty seconds and asks the engine directly: did anything finish that I do not know about? Seven real recoveries have happened in production since launch, averaging 66 seconds. Anything that fails outright lands in a ledger someone can actually inspect, instead of vanishing.
14 min → 66s
I froze everything that changes what users receive
The AI providers still account for most of the runtime. Making them faster would mean changing models, prompts, reasoning depth, or output limits, and therefore changing what users receive. I froze all of those variables. Every measured improvement came from removing operational delay.
Reasoning-effort reduction
Would cut text latency materially. Also risks prompt quality. Not our call to make.
Model swaps
Different behaviour, different output. That is a migration with its own risk budget.
Prompt rewriting or merging steps
Changes the creative construction users already rely on.
Output token caps
Cheap latency win, real truncation risk.
Parallelising the visual chain
Later steps consume earlier steps' images. The dependency is real.
Prompt compression changes
Fires on 0.18% of runs. Not worth the behavioural risk.
Typical, slow, and recovery cases all improved
End to end: the moment a user asks, to the moment the finished media exists. Measured across 888 runs before and 314 after.
Typical run
median · p50
−40s · −11.8%
Slowest 10% of runs
p90, where the waiting had piled up
−87s · −13.3%
Average run
mean of all completions
−35s · −9.4%
Wait before a step starts
ready → dispatched, first attempt
96% within 5s
A lost result surfaces in
7 real recoveries in production
−87%
Time to process a completion
notice received → run advanced
via middleman~200ms
hop removed
Being charged twice for one step
the risk atomic claiming removes
possibleprevented
0 since launch
What users receive
prompts, models, output quality
frozenfrozen
unchanged
The honest caveat: the after window is 314 runs over three days against a 888-run baseline week, and run mix varies. A day heavy on 4K video reads slower no matter what the plumbing does. Early single-day samples looked better (−18% / −23%); the settled multi-day figures above are the conservative ones. The result is directionally strong, but the after sample is still growing.
Nothing shipped switched on
All the code went to production doing nothing. Then each behaviour was turned on by itself, watched by an automated probe for twenty minutes, and only then was the next one enabled. Every layer had its own rollback flag, so any stage could be undone without touching the others.
The rollout, stage by stage
Open a stage to see what it changed and what had to go green before the next one.
3 services · 4 queues · 3 migrations
New queues created, database migrated, all three services deployed, with every new behaviour switched off. Production ran exactly as before. This is the step that makes the rest safe.
verified · ~1,850 tests green across services, dry-run deploys clean
1 PR in the generation engine
Provider dispatch, completion notices, and status polls each get their own lane, and the five-second delivery floor is removed. A short message can no longer queue behind a ninety-second video job.
verified · 20-minute watch · 20/20 green · 0 dead letters
1 PR in the workflow orchestrator
Orchestrator-to-engine calls move off the public internet and onto an internal binding. This is the change that halved step pickup time.
verified · first-attempt pickup 3.5s → 1.6s average, 100% within 5s
2 PRs in the generation engine
Completion notices go straight from engine to orchestrator instead of being relayed by the backend. Rolled out to 10% first, selected deterministically by generation id, then to 100% once the auth path was proven.
verified · 20/20 green at both steps · completion processing ~200ms
2 PRs in the API backend
With no traffic left on it for three days, the relay was switched off, watched, and then deleted along with its feature flag and six now-obsolete tests.
verified · rollout complete · every flag live · 0 pending dead letters
What landed where
Three services had to change together, and stay deployable in any order.
the workflow orchestrator
Owns the run: which step is next, what it costs, when it is done.
- Per-step rows with atomic claiming, engine service binding, minute-reconciler
- Switch dispatch onto the internal binding
- Acknowledge late duplicate completions instead of retrying them into the dead-letter ledger
3 PRs · +1,243 / −233 across 22 files · 2 migrations
the generation engine
Routes each step to a provider and survives their failures.
- Three queue lanes, shared dead-letter ledger, per-message acknowledgement
- Turn the lanes on in production
- Direct completions at 10% with a deterministic canary, then to 100%
4 PRs · +1,772 / −163 across 35 files · 917 tests green
the API backend
The app users actually talk to. Was the middleman.
- Pin the completion payload to a shared contract fixture across all three services
- Stop relaying workflow completions
- Delete the relay code once it had proven unused
3 PRs · 790 tests green · relay removed after 3 stable days