In the last post, sixty-four cameras fed one GPU and came out the other end as a stream of bounding boxes, track IDs, and timestamps. That answers one kind of question well: how many, where, and when. It answers a completely different kind of question badly: find me every clip that looks like this one. Those are not the same problem wearing different syntax. They need different indexes, and a system that only built one of them will still return an answer to the question it can't actually handle. It just won't be a good one.

There is a third kind too: questions phrased in plain language about things nobody thought to track as a structured field, like "was there an unattended bag near the entrance in the last hour." That one wants neither a count nor a resemblance. It wants a description.

Three questions, three indexes

Each path starts from the same sixty-four camera feeds and ends at a completely different data structure, built for a completely different shape of question.

The count, answered exactly

-- a pull query against the same materialized table from the last post
SELECT COUNT(DISTINCT track_id) AS occupancy
FROM current_zone_state
WHERE zone_id = 'zone-3';

This answer is either right or it isn't. There is no partial credit, no ranking, no "close enough." It is also blind to anything the detector was never trained to notice.

The resemblance, answered by ranking

float[] queryVector = embeddingClient.embed(referenceClip);

List<ClipMatch> matches = vectorIndex.search(
    queryVector,
    10,                 // top-K
    DistanceMetric.COSINE
);

matches.stream()
    .filter(m -> m.score() > SIMILARITY_THRESHOLD)
    .forEach(m -> investigationQueue.add(m.clipId()));

This answer is a ranked list, not a fact. The tenth result might be a genuine match or might just be the least-bad option in a mediocre field, and the query has no way of telling you which.

The description, answered in language

SELECT clip_id, camera_id, caption, event_time
FROM camera_captions
WHERE caption ILIKE '%unattended bag%'
  AND event_time > now() - INTERVAL '1' HOUR;

This one only finds what the model chose to write down. If it captioned the bag as "a package near the door," a literal keyword match on "unattended" misses it entirely, which is its own argument for embedding the captions too, not just storing them as text.

"A vector search will always give you its ten best guesses. It will never tell you that none of them were actually right."

A lesson learned in production, not in a demo

Four things that broke without raising an exception

The exact question, asked of the wrong index. Someone needed to know how many people were in a zone right now, and the only search box on the dashboard was the similarity one. It returned ten clips that looked like a crowded zone, ranked by resemblance, none of them an actual count. Nothing failed. The question was just never answerable that way.

The event that fell between two chunks. Embeddings are generated per ten-second window. An incident that starts at second nine and resolves at second twelve gets split across two chunks, each showing only half of it, and each too weak on its own to rank highly against a query built from a clean, complete example. The incident happened. Neither embedding looks enough like it to be found.

Full captions turned on everywhere, quietly starving the fleet. A vision-language model asked to write a full caption for every chunk does meaningfully more work per stream than one asked to emit a single alert token, and that gap directly limits how many concurrent streams a GPU can actually keep up with. Turning on rich descriptions across all sixty-four cameras without re-checking that ceiling means some cameras start falling behind, and the first symptom is stale captions, not an error anyone notices immediately.

Yesterday's clips, embedded in a language today's model no longer speaks. Swapping in an upgraded embedding model without re-embedding the historical index leaves old and new vectors sitting in the same similarity space with different underlying geometry. Cosine similarity between them is not meaningless, but it is not comparable either, so day-old footage quietly stops matching queries it should have matched, and nothing about the query looks wrong.

Which question goes where

The honest number

Across a week of real queries against the fleet, roughly six in ten were the exact kind, answerable by the structured table alone. Three in ten needed the similarity index. The last one in ten only made sense as a plain-language question, and no amount of clever SQL or better embeddings would have answered it, because the thing being asked for was never a field, a vector, or a threshold. It was a description.

"The camera never asks you which kind of question you meant. Building three indexes instead of one is how you answer it anyway."

Earned opinion, not a benchmark leaderboard