import org.apache.spark.sql.streaming.OutputMode.Update val inputStream = spark .readStream .format("rate") .load .writeStream .format("console") .outputMode(Update) // <-- update output mode .start
Output mode (
OutputMode) of a streaming query describes what data is written to a streaming sink.
There are three available output modes:
The output mode is specified on the writing side of a streaming query using DataStreamWriter.outputMode method (by alias or a value of
Append (alias: append) is the default output mode that writes "new" rows only.
In streaming aggregations, a "new" row is when the intermediate state becomes final, i.e. when new events for the grouping key can only be considered late which is when watermark moves past the event time of the key.
Append output mode requires that a streaming query defines event-time watermark (using withWatermark operator) on the event time column that is used in aggregation (directly or using window function).
Required for datasets with
FileFormat format (to create FileStreamSink)
Append is mandatory when multiple
flatMapGroupsWithState operators are used in a structured query.
Complete (alias: complete) writes all the rows of a Result Table (and corresponds to a traditional batch structured query).
Complete mode does not drop old aggregation state and preserves all data in the Result Table.