InputPartitionReader Contract

InputPartitionReader is the abstraction of input partition readers in Data Source API V2 that can proceed to the next record and get the current record.

InputPartitionReader is also a Java Closeable.

InputPartitionReader is associated with two other abstractions: InputPartition and ContinuousInputPartition that are responsible for creating concrete InputPartitionReaders.

Note
 It appears that all concrete input partition readers are used in Spark Structured Streaming only.
Table 1. InputPartitionReader Contract
Method Description

get

 
T get()

Gets the current record

Used when:

  • DataSourceRDD is requested to compute a partition

  • DataReaderThread is requested to run (start up)

next

Proceeds to the next record if available (true)

Used when:

  • DataSourceRDD is requested to compute a partition

  • DataReaderThread is requested to run (start up)
Table 2. InputPartitionReaders (Direct Implementations and Extensions Only)
InputPartitionReader Description

ContinuousInputPartitionReader

Extension that is used in Spark Structured Streaming for Continuous Stream Processing

KafkaMicroBatchInputPartitionReader

Used in Spark Structured Streaming for Kafka Data Source

Anonymous

Used in Spark Structured Streaming for Memory Data Source

RateStreamMicroBatchInputPartitionReader

Used in Spark Structured Streaming for Rate Data Source

Anonymous

Used in Spark Structured Streaming for Text Socket Data Source

results matching ""

    No results matching ""