void commit(Offset end)
MicroBatchReader Contract — Data Source Readers in Micro-Batch Stream Processing (Data Source API V2)
MicroBatchReader
is the extension of Spark SQL’s DataSourceReader
(and BaseStreamingSource) contracts for data source readers in Micro-Batch Stream Processing.
MicroBatchReader
is part of the novel Data Source API V2 in Spark SQL.
Tip
|
Read up on Data Source API V2 in The Internals of Spark SQL book. |
Method | Description |
---|---|
|
Used when…FIXME |
|
|
|
|
|
|
|
Sets the desired offset range for input partitions created from this reader (for data scan) Used when…FIXME |
MicroBatchReader | Description |
---|---|
RateStreamMicroBatchReader |
|
TextSocketMicroBatchReader |