InputPartition Contract

InputPartition is also a Java Serializable.

InputPartition is associated with DataSourceReader abstraction and its extension SupportsScanColumnarBatch.

InputPartition is used for the following:

Note
It appears that all concrete input partitions are used in Spark Structured Streaming only.
Table 1. InputPartition Contract
Method Description

createPartitionReader

InputPartitionReader<T> createPartitionReader()

Used when:

preferredLocations

String[] preferredLocations()

Specifies the preferred locations (executor hosts)

Default: (empty)

Used when:

  • DataSourceRDD is requested for the preferred locations

  • ContinuousDataSourceRDD is requested for the preferred locations

Table 2. InputPartitions (Direct Implementations and Extensions Only)
InputPartition Description

ContinuousInputPartition

InputPartitions for Continuous Stream Processing in Spark Structured Streaming

Consult The Internals of Spark Structured Streaming

ContinuousMemoryStreamInputPartition

Used in Spark Structured Streaming

KafkaMicroBatchInputPartition

Used in Spark Structured Streaming

MemoryStreamInputPartition

Used in Spark Structured Streaming

RateStreamMicroBatchInputPartition

Used in Spark Structured Streaming

TextSocketContinuousInputPartition

Used in Spark Structured Streaming

Anonymous

Used in Spark Structured Streaming

results matching ""

    No results matching ""