Powered by GitBook

InputPartition Contract

InputPartition is the abstraction of input partitions in Data Source API V2 that can create an InputPartitionReader and optionally specify preferred locations.

InputPartition is also a Java Serializable.

InputPartition is associated with DataSourceReader abstraction and its extension SupportsScanColumnarBatch.

InputPartition is used for the following:

Creating DataSourceRDD, DataSourceRDDPartition, ContinuousDataSourceRDD and ContinuousDataSourceRDDPartition
Requesting DataSourceV2ScanExec physical operator for the partitions or batchPartitions (of the input RDD)

Note	It appears that all concrete input partitions are used in Spark Structured Streaming only.

Table 1. InputPartition Contract
Method	Description
`createPartitionReader`	`InputPartitionReader<T> createPartitionReader()` Creates an InputPartitionReader Used when: `DataSourceRDD` is requested to compute a partition `ContinuousQueuedDataReader` is created
`preferredLocations`	`String[] preferredLocations()` Specifies the preferred locations (executor hosts) Default: `(empty)` Used when: `DataSourceRDD` is requested for the preferred locations `ContinuousDataSourceRDD` is requested for the preferred locations

Table 2. InputPartitions (Direct Implementations and Extensions Only)
InputPartition	Description
`ContinuousInputPartition`	InputPartitions for Continuous Stream Processing in Spark Structured Streaming Consult The Internals of Spark Structured Streaming
`ContinuousMemoryStreamInputPartition`	Used in Spark Structured Streaming
`KafkaMicroBatchInputPartition`	Used in Spark Structured Streaming
`MemoryStreamInputPartition`	Used in Spark Structured Streaming
`RateStreamMicroBatchInputPartition`	Used in Spark Structured Streaming
`TextSocketContinuousInputPartition`	Used in Spark Structured Streaming
`Anonymous`	Used in Spark Structured Streaming

results matching ""

No results matching ""