InputPartition Contract

Table 1. InputPartition Contract
Method Description

createPartitionReader

InputPartitionReader<T> createPartitionReader()

Used when:

preferredLocations

String[] preferredLocations()

Specifies the preferred locations (executor hosts)

Default: (empty)

Used when:

  • DataSourceRDD is requested for the preferred locations

  • ContinuousDataSourceRDD is requested for the preferred locations

Table 2. InputPartitions (Direct Implementations)
InputPartition Description

Anonymous

Used in Spark Structured Streaming’s TextSocketMicroBatchReader

ContinuousInputPartition

ContinuousMemoryStreamInputPartition

Used in Spark Structured Streaming

KafkaMicroBatchInputPartition

Used in Spark Structured Streaming

MemoryStreamInputPartition

Used in Spark Structured Streaming

RateStreamMicroBatchInputPartition

Used in Spark Structured Streaming

TextSocketContinuousInputPartition

Used in Spark Structured Streaming

results matching ""

    No results matching ""