MicroBatchReader Contract — Data Source Readers in Micro-Batch Stream Processing (Data Source API V2)

MicroBatchReader is the extension of Spark SQL’s DataSourceReader (and BaseStreamingSource) contracts for data source readers in Micro-Batch Stream Processing.

MicroBatchReader is part of the novel Data Source API V2 in Spark SQL.

Tip
 Read up on Data Source API V2 in The Internals of Spark SQL book.
Table 1. MicroBatchReader Contract
Method Description

commit

 
void commit(Offset end)

Used when…​FIXME

deserializeOffset

 
Offset deserializeOffset(String json)

Deserializes offset (from JSON format)

Used when…​FIXME

getEndOffset

 
Offset getEndOffset()

End offset of this reader

Used when…​FIXME

getStartOffset

 
Offset getStartOffset()

Start (beginning) offsets of this reader

Used when…​FIXME

setOffsetRange

 
void setOffsetRange(
  Optional<Offset> start,
  Optional<Offset> end)

Sets the desired offset range for input partitions created from this reader (for data scan)

Used when…​FIXME
Table 2. MicroBatchReaders
MicroBatchReader Description

KafkaMicroBatchReader

MemoryStream

RateStreamMicroBatchReader

TextSocketMicroBatchReader

results matching ""

    No results matching ""