Streaming Data
Some data arrives like a finished spreadsheet you can download and analyze. Other data shows up like a never-ending ticker—new rows keep appearing, and you’re expected to react while it’s still coming in. That second kind is what people mean by streaming data.
What “streaming” means
Streaming data is data generated and delivered continuously (or in frequent bursts) over time, often in time order. Instead of having a fixed-size dataset, you have an ongoing sequence of events or observations—for example, one purchase, one sensor reading, or one app click at a time. A key practical constraint is that you typically can’t store or repeatedly scan the entire history, so you process data incrementally as it arrives.
Everyday examples
Common sources of streaming data include:
- House price listings updating throughout the day (new listings, price cuts).
- Exam platform logs as students submit answers in real time.
- Retail transactions arriving from point-of-sale systems.
- Medical monitors producing heart rate or oxygen readings every second.
Why it matters in statistics and ML
Streaming settings change what “analysis” looks like. You often rely on:
- Online/streaming statistics (running mean/variance, quantiles via sketches) rather than batch recomputation.
- Windowing (last 5 minutes, last 10,000 events) to focus on recent behavior.
- Concept drift handling, because the data-generating process can change (fraud patterns, user tastes, sensor calibration).
In ML, streaming enables near-real-time anomaly detection, recommendations, and monitoring, but it also makes evaluation and training trickier: you must avoid “peeking into the future” and often use prequential (test-then-train) evaluation.
Where you’ll see it
Streaming pipelines commonly use Kafka/Kinesis for ingestion and Spark Structured Streaming/Flink for processing; online learning appears in libraries like River (formerly creme).
Streaming Data is data generated and delivered continuously over time, often as an unbounded sequence of events rather than a fixed dataset. It typically requires online processing: updating statistics or model parameters incrementally as new observations arrive, sometimes under latency and memory constraints. It matters in AI/ML because many systems must learn and detect changes in real time. Example: monitoring clickstream events to update conversion-rate estimates and trigger anomaly alerts.
Imagine watching a live sports game: the action keeps coming, and you can’t pause the world to collect the whole match before reacting. That’s like streaming data—information that arrives continuously over time, often in small pieces.
In statistics and AI/ML, streaming data might be sensor readings from a smartwatch, clicks on a website, or transactions in a bank. Because it never really “finishes,” systems often analyze it as it arrives, updating charts, spotting unusual activity, or adjusting predictions in near real time instead of waiting for a complete dataset.