ContinuousReader Contract — Continuous Source in Continuous Stream Processing

ContinuousReader is the extension of Spark SQL’s DataSourceReader (and BaseStreamingSource) contracts for data source readers that manage offsets in Continuous Stream Processing.

Table 1. ContinuousReader Contract
Method Description

commit

void commit(Offset end)

Commits the specified offset

Used exclusively when ContinuousExecution is requested to commit an epoch

deserializeOffset

Offset deserializeOffset(String json)

Deserializes an offset from JSON representation

Used when ContinuousExecution is requested to run a streaming query and commit an epoch

getStartOffset

Offset getStartOffset()
Note
Used exclusively in tests.

mergeOffsets

Offset mergeOffsets(PartitionOffset[] offsets)

Used exclusively when ContinuousExecution is requested to addOffset

needsReconfiguration

boolean needsReconfiguration()

Used exclusively when ContinuousExecution is requested to run the streaming query in continuous mode.

setStartOffset

void setStartOffset(Optional<Offset> start)

Used exclusively when ContinuousExecution is requested to run the streaming query in continuous mode.

Table 2. ContinuousReaders
ContinuousReader Description

ContinuousMemoryStream

KafkaContinuousReader

RateStreamContinuousReader

TextSocketContinuousReader

results matching ""

    No results matching ""