Ask someone to sketch a real-time vision system and they draw a camera, an arrow, and a model. That arrow is doing an enormous amount of unpaid work. A single 1080p stream decoded on a CPU core burns thirty to forty percent of it before a single pixel reaches anything resembling intelligence. Multiply that by sixty-four cameras and the model was never going to be the bottleneck. Getting pixels to it, fast enough and in the right shape, was always the actual engineering problem.

I wanted to see what a real deployment of that arrow looks like: sixty-four camera streams, one GPU, a detection model, and a latency budget tight enough that the output has to be a decision, not a highlight reel someone reviews the next morning.

The layered pipeline

Every stage exists to keep the GPU fed and to make sure what comes out the other end is a fact, not a frame. Sixty-four cameras never mean sixty-four separate pipelines. They mean sixty-four sources feeding one shared set of stages.

The stages, as a pipeline graph

rtsp-source-0 ..63 -> nvstreammux (batch=64)
             -> nvinfer (TensorRT INT8 detector)
             -> nvtracker (per-object identity)
             -> nvmsgconv (frame metadata -> JSON schema)
             -> nvmsgbroker (publish -> kafka topic "camera.detections")

Everything to the left of the batching stage is per-camera. Everything from batching onward is shared. That single line is where sixty-four separate video feeds stop being sixty-four separate problems.

Turning the metadata into a decision, in Kafka Streams

KStream<String, Detection> detections = builder.stream("camera.detections");

detections
    .filter((key, d) -> d.objectClass().equals("person"))
    .groupBy((key, d) -> d.cameraId(), Grouped.with(Serdes.String(), detectionSerde))
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofSeconds(10)))
    .aggregate(
        HashSet::new,
        (key, d, ids) -> { ids.add(d.trackId()); return ids; },
        Materialized.with(Serdes.String(), trackIdSetSerde)
    )
    .toStream()
    .filter((windowedKey, ids) -> ids.size() > ZONE_CAPACITY)
    .to("camera.zone-alerts");

The camera never enters this code. By the time a decision gets made, the input is a set of track IDs in a window, indistinguishable in shape from any other stream of structured events this blog has covered.

"A bounding box is a shape. A track ID is a claim. The gap between them is where every real bug in this architecture lives."

A lesson learned in production, not in a demo

Four things that broke without raising an exception

The batch-wait tax. Batching sixty-four streams into one inference call means the batch cannot run until the slowest stream's frame arrives. One camera on a congested network link does not just run slower itself, it drags down the effective latency of every other stream sharing that batch. Documented multi-camera deployments show exactly this pattern: some sources sitting comfortably under 150ms while others spike past 400ms in the same batch, with no single component reporting a failure.

The silent fallback to software decode. One camera streaming a codec profile the hardware decoder does not support quietly falls back to CPU decode instead of failing outright. That single stream now burns a full core, competes with everything else for CPU time, and every dashboard still shows it as a healthy, connected source.

Track ID churn across an occlusion. A person walks behind a shelf for half a second and the tracker assigns a new ID when they reappear. Anything counting unique people, vehicles, or defects downstream just silently double-counted one of them. Nothing errors. The count is just wrong, by an amount nobody can see without auditing individual tracks.

The operator that spills to CPU. An updated model ships with one layer the inference engine does not have an optimized kernel for. That single operator falls back to CPU execution mid-graph, and average latency creeps upward with no crash, no log line calling it out, just a slow, cumulative tax on every frame that touches it.

Where the GPU actually sits

Sixty-four streams into one GPU is a centralization decision, and it is not the only one available. The other end of the spectrum pushes inference onto a small device sitting next to each camera.

The honest number

At sixty-four streams and fifteen frames per second, the pipeline held a p99 end-to-end latency under 150 milliseconds, decode through published decision, on a single GPU. The number that mattered more was zero: zero frames of raw video ever written to disk, because the only thing that survived past the tracker was a bounding box, a class, and an ID. The camera was never the product. The stream of facts it produced was.

"Nobody needs the video. They need to know, in time to act, what was in it."

Earned opinion, not a benchmark leaderboard