StreamWriter Contract

StreamWriter is the extension of the DataSourceWriter contract to support epochs, i.e. streaming writers that can abort and commit writing jobs for a specified epoch.

Tip
 Read up on DataSourceWriter in The Internals of Spark SQL book.
Table 1. StreamWriter Contract
Method Description

abort

 
void abort(
  long epochId,
  WriterCommitMessage[] messages)

Aborts the writing job for a specified epochId and WriterCommitMessages

Used exclusively when MicroBatchWriter is requested to abort

commit

 
void commit(
  long epochId,
  WriterCommitMessage[] messages)

Commits the writing job for a specified epochId and WriterCommitMessages

Used when:

  • EpochCoordinator is requested to commitEpoch

  • MicroBatchWriter is requested to commit
Table 2. StreamWriters
StreamWriter Description

ForeachWriterProvider

foreachWriter data source

ConsoleWriter

console data source

KafkaStreamWriter

kafka data source

MemoryStreamWriter

memory data source

results matching ""

    No results matching ""