Sink Contract — Streaming Sinks for Micro-Batch Stream Processing

Sink is the extension of the BaseStreamingSink contract for streaming sinks that can add batches to an output.

Sink is part of Data Source API V1 and used in Micro-Batch Stream Processing only.

Table 1. Sink Contract
Method Description

addBatch

 
addBatch(
  batchId: Long,
  data: DataFrame): Unit

Adds a batch of data to the sink

Used exclusively when MicroBatchExecution stream execution engine (Micro-Batch Stream Processing) is requested to add a streaming batch to a sink (addBatch phase) while running an activated streaming query.
Table 2. Sinks
Sink Description

FileStreamSink

Used in file-based data sources (FileFormat)

ForeachBatchSink

Used for DataStreamWriter.foreachBatch streaming operator

KafkaSink

Used for kafka output format

MemorySink

Used for memory output format

results matching ""

    No results matching ""