Structured Streaming
E705278
Structured Streaming is Apache Spark’s scalable, fault-tolerant stream processing engine that lets developers express streaming computations using the same high-level APIs as batch processing.
Statements (49)
| Predicate | Object |
|---|---|
| instanceOf |
Apache Spark component
ⓘ
stream processing engine ⓘ |
| APIStyle |
declarative
ⓘ
unified batch and streaming API ⓘ |
| designedFor | exactly-once processing with idempotent sinks ⓘ |
| developedBy | Apache Software Foundation NERFINISHED ⓘ |
| documentationURL | https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html ⓘ |
| exposesAPI |
DataFrame API
ⓘ
Dataset API ⓘ Spark SQL API NERFINISHED ⓘ |
| integratesWith |
Spark MLlib
NERFINISHED
ⓘ
Spark SQL NERFINISHED ⓘ Spark Structured APIs NERFINISHED ⓘ |
| introducedIn | Apache Spark 2.0 NERFINISHED ⓘ |
| partOf | Apache Spark NERFINISHED ⓘ |
| provides |
backpressure handling
ⓘ
end-to-end event-time processing ⓘ exactly-once semantics (under certain conditions) ⓘ fault tolerance ⓘ stateful stream processing ⓘ watermarking for late data ⓘ windowed aggregations ⓘ |
| replaced | DStreams for many use cases ⓘ |
| stores |
offsets
ⓘ
state in state store ⓘ streaming query progress metadata ⓘ |
| supports |
checkpointing
ⓘ
continuous processing ⓘ event-time windows ⓘ micro-batch processing ⓘ near real-time data processing ⓘ session windows ⓘ sliding windows ⓘ stream processing ⓘ |
| supportsMode |
append output mode
ⓘ
complete output mode ⓘ update output mode ⓘ |
| supportsSink |
Kafka sink
ⓘ
console sink ⓘ file sink ⓘ foreach sink ⓘ memory sink ⓘ |
| supportsSource |
Kafka
NERFINISHED
ⓘ
file source ⓘ rate source ⓘ socket source ⓘ |
| uses |
Catalyst optimizer
NERFINISHED
ⓘ
Spark SQL engine NERFINISHED ⓘ Tungsten execution engine NERFINISHED ⓘ |
Referenced by (1)
Full triples — surface form annotated when it differs from this entity's canonical label.