ContinuousReader Contract — Data Source Readers in Continuous Stream Processing

ContinuousReader is the extension of Spark SQL’s DataSourceReader (and BaseStreamingSource) contracts for data source readers in Continuous Stream Processing.

ContinuousReader is part of the novel Data Source API V2 in Spark SQL.

Tip	Read up on Data Source API V2 in The Internals of Spark SQL book.

Method Description

commit

void commit(Offset end)

Commits the specified offset

Used exclusively when ContinuousExecution is requested to commit an epoch

deserializeOffset

Offset deserializeOffset(String json)

Deserializes an offset from JSON representation

Used when ContinuousExecution is requested to run a streaming query and commit an epoch

getStartOffset

Offset getStartOffset()

Note	Used exclusively in tests.

mergeOffsets

Offset mergeOffsets(PartitionOffset[] offsets)

Used exclusively when ContinuousExecution is requested to addOffset

needsReconfiguration

boolean needsReconfiguration()

Indicates that the reader needs reconfiguration (e.g. to generate new input partitions)

Used exclusively when ContinuousExecution is requested to run a streaming query in continuous mode

setStartOffset

void setStartOffset(Optional<Offset> start)

Used exclusively when ContinuousExecution is requested to run the streaming query in continuous mode.

Table 2. ContinuousReaders
ContinuousReader	Description
ContinuousMemoryStream
KafkaContinuousReader
RateStreamContinuousReader
TextSocketContinuousReader

ContinuousReader Contract