Computer Vision Is a Batch Job

Start with the design every tutorial for serving a model teaches, and grant it everything, because it's a good design and the rest of this depends on taking it seriously. Put the model behind an HTTP endpoint. Give the endpoint a GPU. Autoscale on request rate, watch what the slowest requests cost you, and when traffic doubles, add replicas. The model is now a service, with a promise attached about how fast it answers, and most of what you already know about running services — health checks, canary deploys, a dashboard where a slow tail shows up — transfers to it intact.

That design is right whenever something is waiting on the answer. A camera at a gate that has to decide in 200 milliseconds whether to open it. A moderation check between a user's upload and their followers' feeds. A radiologist with the study open. In all of those, latency is the product, and you should pay for idle GPUs the way a restaurant pays for a lit dining room at four in the afternoon.

Then there is the other kind of work, which is most of the computer vision I've been paid to build. A microscope finishes a plate run at two in the morning and leaves nine thousand images in a bucket. A drone lands and its card is copied to a laptop. A scanner works through a shelf of slides. The images arrive in a heap, they arrive at an hour nobody chose, and the person who will read the results is asleep and will read them at nine.

Nothing is waiting. And once nothing is waiting, the endpoint is the wrong shape — wrong in a way that shows up as a line item. Most computer vision inference is not a service. It's a batch job, and the money is in admitting it.

Latency you aren't spending

A service spends its latency budget the moment a request arrives: the clock is running, someone is looking at a spinner, and every architectural decision after that is about getting the answer back before they give up. A batch job is handed the same budget and is never asked for it. The plate images have until nine in the morning. That's seven hours of slack that the endpoint design throws away by being ready.

Latency you don't spend is money you haven't collected yet, and the exchange rate is published. AWS sells interruptible capacity — Spot — at up to a ninety percent discount against On-Demand.¹ The hardware is the same hardware. What you're selling back for the discount is the guarantee that you get to keep the instance: AWS can take it, with two minutes' notice, whenever it wants the capacity for someone paying full price.²

For a service, that's an outage, and no discount makes it worth taking. For work with seven hours of slack, it's a schedule. The job can be killed and rerun. It can wait for capacity in a different availability zone, or a different instance family, or for twenty minutes. It can start cold. None of those cost the biologist anything, because all of them finish before breakfast, and each of them is a thing the endpoint could not do and therefore could not sell.

That's the trade. What remains is the accounting on it: what it costs to hold work that can be interrupted, and whether the discount survives what it costs.

The arithmetic of a batch

The first cost is the model. A large segmentation model takes something like thirty to sixty seconds to load into GPU memory, and a worker pays that once and then runs. Spread over a job of ten images, a forty-five second load is four and a half seconds of waste per image, which is about what the inference itself costs. Spread over a thousand images, it disappears. So far, bigger jobs win.

The second cost pulls the other way. A worker that is interrupted redoes whatever it hasn't recorded, and a longer job has more to lose — it's exposed for longer, and there's more of it to lose when the notice comes. Take the simplest design, where a job restarts from its beginning, and those two costs have a crossing point.

Under one plausible setting of the numbers — forty-five seconds to load, five seconds of compute per image, an interruption rate of twenty percent per worker-hour — the cheapest batch is 255 images.³ The useful part of that chart is not the 255. It's the shape of the bottom: anything between 163 and 399 images lands within a tenth of the best you can do, and the walls are steep outside it. Jobs of ten are wasteful and jobs of ten thousand are reckless, and in between there's a wide, forgiving basin where the exact number doesn't matter much.

Which is why a few hundred images per job is a defensible answer even though it looks like folklore. It isn't a tuned constant. It's the middle of a basin — and the walls are what move when your numbers do.

Checkpoints are what the two minutes are for

Look again at the right-hand wall. It's made entirely of work that was already done and got thrown away, which means it's the wall you can move.

Write results as they're produced rather than at the end, record the index of the last image finished, and on startup read that index and resume. Now an interruption costs the images since the last checkpoint instead of the whole job, and the redo term in that arithmetic stops being about the size of the job at all. It's about the checkpoint interval. Jobs can get longer; the basin widens to the right; the discount survives.

This is what the two-minute notice is actually for. It arrives as an EventBridge event and in the instance metadata, and a worker that listens for it can finish the image it's on, flush its checkpoint, and requeue the remainder before the instance goes away. A worker that doesn't listen for it gets killed mid-image and redoes the interval anyway, which is survivable — the listener is an optimization on top of the checkpoint, not a substitute for it. The checkpoint is what makes interruption ordinary, and it gives you the one line worth writing on a whiteboard: the work between two checkpoints is the work you're willing to do twice.

Checkpointing has a cost of its own, so it doesn't go to zero: every checkpoint is a small write, and at one per image on nine thousand images you've added nine thousand round trips to a job that could have done nine hundred. The same shape as before, one level down.

The GPU that isn't computing

A job that finished is not the same as a job that ran well, and job completion is the metric that can't tell the difference.

You rent the instance by the wall-clock second, and that meter runs through everything the worker does: pulling images down, decoding JPEGs, resizing, running the model, encoding masks, writing them back. Only one of those needs the GPU. During the rest of them the GPU is idle and billed, and a pipeline that pulls one image, decodes it, infers, writes the result, and then pulls the next one can leave a very expensive card idle for more of the hour than it spends computing. Every job completes. Every dashboard is green. And the GPU-hours on the invoice are a multiple of the GPU-seconds that did the work.

Utilization sees this and completion doesn't, so the thing to put on the wall is the GPU's own number. Then the fixes are the ordinary ones: fetch and decode the next batch on worker threads while the current one is on the GPU, write results asynchronously, size the batch to the memory rather than to a habit, and move preprocessing that never needed a GPU onto instances that don't have one.

You may be about to say that a GPU at forty percent can be perfectly fine, and that's true. If the job is short, or runs once a week, or the engineering time to fix it costs more than the instance hours it saves, forty percent is a decision. The failure isn't a low number; it's not having the number, and finding out from the bill.

What that makes you build

None of the components are clever, and that's the argument for them: each one is what's left after you stop pretending someone is waiting.

A queue holds the unit of work, because the unit of work is now a durable thing rather than a request in flight. A job is a list of image keys and a model version. It can sit in the queue for an hour, be delivered to a worker that dies, and come back.

The orchestrator and the inference code don't know about each other. The orchestrator scales workers, retries, requeues, and enforces concurrency; it knows nothing about segmentation. The worker loads a model and runs it; it knows nothing about spot capacity. That line is what lets a model change without an infrastructure change, and — more often, in practice — an infrastructure change without re-testing the model.

A model registry: versioned artifacts in object storage, a small table pointing at them, and every result row carrying the version that produced it. The images I've worked on came from biological research, where the question which model produced this number gets asked months later by someone who wasn't in the room, and that field is the whole answer to it. It costs nothing to write and cannot be reconstructed after the fact.

And the cost of each job recorded with the job: instance type, seconds, the price actually paid. A dollar figure per plate, available the day it ran rather than on the invoice at the end of the month. This is the one people skip, and it's the one that makes every other decision here arguable instead of theological.

That's four boxes and a queue. It's not a framework and it doesn't want to be one. It's what the batch shape asks for, and you can build it in two weeks on services you're already paying for.

Who is waiting

The endpoint design is still right, and if the camera at the gate is yours, build it. The mistake isn't the architecture. It's inheriting it — reaching for a GPU that's always on because that's what the tutorial had, for work whose only real deadline is a person's morning.

So the first question about a vision model in production isn't how fast it answers. It's who is waiting for the answer. If it's nobody until nine, then those hours aren't a constraint you have to work around. They're the cheapest thing you own, and the only reason they're on the bill at full price is an architecture that assumed somebody was there.

Footnotes

1. Amazon Web Services, Amazon EC2 Spot Instances (aws.amazon.com/ec2/spot/): Spot Instances "are available at up to a 90% discount compared to On-Demand prices," as the page read in September 2026. Up to is the ceiling and not the expectation; the discount varies by instance type, by region, and by the hour. ↩

2. Amazon EC2 User Guide, Spot Instance interruption notices: EC2 issues the notice two minutes before it interrupts the instance, delivered as an Amazon EventBridge event and as an item in the instance metadata. A separate rebalance recommendation can arrive earlier, when an instance is at elevated risk of interruption, though it can also arrive alongside the two-minute notice rather than before it — as the documentation read in September 2026. ↩

3. The parameters are illustrative and the curve is arithmetic rather than a measurement: 45 seconds to load the model, 5 seconds of compute per image, an interruption hazard of 20 percent per worker-hour. Overhead per image is L/n + λ(L + nt)²/2n — the model load spread across the job, plus the expected redo when a job of duration L + nt is interrupted and restarts from its beginning. The next section replaces that n in the second term with the checkpoint interval, which is the whole point of checkpointing. The script that draws the chart is checked in beside this piece; change the three numbers and the basin moves, which is what it is for. ↩