ContinuousReader Contract — Data Source Readers in Continuous Stream Processing

ContinuousReader is the extension of Spark SQL’s DataSourceReader (and BaseStreamingSource) contracts for data source readers in Continuous Stream Processing.

ContinuousReader is part of the novel Data Source API V2 in Spark SQL.

Table 1. ContinuousReader Contract
Method Description

commit

void commit(Offset end)

Commits the specified offset

Used exclusively when ContinuousExecution is requested to commit an epoch

deserializeOffset

Offset deserializeOffset(String json)

Deserializes an offset from JSON representation

Used when ContinuousExecution is requested to run a streaming query and commit an epoch

getStartOffset

Offset getStartOffset()
Note
Used exclusively in tests.

mergeOffsets

Offset mergeOffsets(PartitionOffset[] offsets)

Used exclusively when ContinuousExecution is requested to addOffset

needsReconfiguration

boolean needsReconfiguration()

Indicates that the reader needs reconfiguration (e.g. to generate new input partitions)

Used exclusively when ContinuousExecution is requested to run a streaming query in continuous mode

setStartOffset

void setStartOffset(Optional<Offset> start)

Used exclusively when ContinuousExecution is requested to run the streaming query in continuous mode.

Table 2. ContinuousReaders
ContinuousReader Description

ContinuousMemoryStream

KafkaContinuousReader

RateStreamContinuousReader

TextSocketContinuousReader

results matching ""

    No results matching ""