Ask a normal database a question and it answers once, from data that already sat still long enough to be counted. Ask the same question of a stream and the honest answer is "as of right now, but keep watching, because the next event might change what I just told you." Most of the trouble in real-time decisioning systems comes from pretending that second sentence isn't there.

I wanted to see what that actually looks like end to end: telemetry pulled straight off networking hardware, landed on Kafka, and queried continuously to decide, in near real time, whether a link is congested enough to reroute traffic before anyone notices.

From a gRPC subscription to a queryable table

Modern top-of-rack switches do not wait to be polled. They speak gNMI, a gRPC-based protocol where a collector subscribes once and the switch pushes updates back, either on a fixed interval or the moment a value changes. gnmic, the open-source collector, does the subscribing and writes every update straight onto a Kafka topic, one partition per switch. Everything downstream, Kafka Streams, ksqlDB, or Flink, just sees an ordinary topic. It has no idea the data came off a switch's ASIC ten milliseconds ago.

Six stages turn that raw feed into something a remediation service can act on.

The windowed aggregation, in Kafka Streams

KStream<String, InterfaceCounter> counters = builder.stream("tor.interface.counters");

counters
    .groupBy((key, v) -> v.interfaceId(), Grouped.with(Serdes.String(), counterSerde))
    .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofSeconds(10), Duration.ofSeconds(2)))
    .aggregate(
        CongestionStats::empty,
        (key, v, agg) -> agg.withDrop(v.bufferDrops()).withDepth(v.queueDepth()),
        Materialized.<String, CongestionStats, WindowStore<Bytes, byte[]>>
            as("congestion-store")
    )
    .toStream()
    .filter((windowedKey, stats) -> stats.dropRate() > THRESHOLD)
    .to("tor.congestion.alerts");

The grace period on that window, two seconds, is the entire watermark discipline in one argument. Anything arriving within it still counts. Anything later does not, unless the topology explicitly says otherwise, and most do not.

The same idea, expressed as SQL in ksqlDB

CREATE TABLE interface_congestion AS
  SELECT interface_id,
         AVG(buffer_drops) AS drop_rate,
         AVG(queue_depth) AS avg_depth
  FROM tor_interface_counters
  WINDOW TUMBLING (SIZE 10 SECONDS)
  GROUP BY interface_id
  EMIT CHANGES;

-- the NOC dashboard subscribes to every update as it happens
SELECT * FROM interface_congestion EMIT CHANGES;

-- the remediation service asks once, gets one answer, and moves on
SELECT drop_rate FROM interface_congestion WHERE interface_id = 'tor-042-eth3';

And in Flink SQL, where the watermark is explicit

CREATE TABLE tor_interface_counters (
  interface_id STRING,
  buffer_drops BIGINT,
  queue_depth BIGINT,
  event_time TIMESTAMP(3),
  WATERMARK FOR event_time AS event_time - INTERVAL '2' SECOND
) WITH ('connector' = 'kafka', 'topic' = 'tor.interface.counters');

SELECT interface_id,
       TUMBLE_START(event_time, INTERVAL '10' SECOND) AS window_start,
       AVG(buffer_drops) AS drop_rate
FROM tor_interface_counters
GROUP BY interface_id, TUMBLE(event_time, INTERVAL '10' SECOND);

Flink makes you write the watermark clause by hand. ksqlDB and Kafka Streams pick a policy for you and let you override it. Neither is wrong. One just makes the trade-off between latency and completeness impossible to ignore.

"A push query tells you what changed. A pull query tells you what's true. A real-time decision usually needs both, and confuses them at its own risk."

A lesson learned in production, not in a tutorial

Four things that broke without raising an exception

The reconnect that stalled every other switch's alerts. gRPC sessions drop, and gnmic resubscribes automatically. But the resubscribe delivers a burst of buffered updates all at once, timestamped for the seconds it was disconnected. If the watermark for that partition can only advance as fast as its slowest event, one flaky switch quietly delays congestion alerts for switches that never had a problem.

The state store that got OOMKilled during a routine rebalance. Kafka Streams keeps windowed aggregates in RocksDB, which runs off-heap. Setting the JVM's heap size does not bound it. When a pod restarts and the state store rebuilds from its changelog topic, memory climbs in a way no heap dashboard shows, right up until Kubernetes kills the container for using memory nobody budgeted for.

The congestion spike that arrived two seconds too late. The whole point of the buffer-drop counter is to catch congestion as it builds. During a reconnect, the exact event that would have crossed the alert threshold lands after its ten-second window has already closed and emitted a clean result. By default it is simply discarded. The metric exists. The alert never fires.

The pull query that answered a question that was no longer true. Pull queries against ksqlDB read from materialized state that is eventually, not immediately, consistent with the writes behind it. The remediation service asked whether an interface was congested, got "no," and moved on, a few seconds after the underlying table had already been updated to "yes." Nothing failed. The answer was just stale by exactly the amount that mattered.

Four ways to answer a question about a stream

There is a real trilemma underneath all of this: freshness, ad-hoc query flexibility, and resource cost. Every engine here picks two.

The honest number

Once the reconnect handling and the pull-query staleness were both fixed, false negatives, real congestion that slipped through unflagged, dropped from 4.1% of labeled incidents to 0.3% on a week of replayed traffic. Push-query latency to the dashboard stayed under 400 milliseconds end to end. Neither number came from a better model or a faster engine. Both came from being honest about what "current" means when the data never stops arriving.

"A stream doesn't owe you a final answer. It owes you the best answer as of now, and the discipline to say when now has changed."

Earned opinion, not a benchmark leaderboard