MicroBatchWriter — Data Source Writer in Micro-Batch Stream Processing (Data Source API V2)
MicroBatchWriter
is a DataSourceWriter
(Spark SQL) that uses the given batch ID as the epoch when requested to commit, abort and create a WriterFactory
for a given StreamWriter in Micro-Batch Stream Processing.
Tip
|
Read up on DataSourceWriter in The Internals of Spark SQL book. |
MicroBatchWriter
is part of the novel Data Source API V2 in Spark SQL.
MicroBatchWriter
is created exclusively when MicroBatchExecution
is requested to run a streaming batch (with a StreamWriteSupport streaming sink).