ShuffleReader

package org.apache.spark.shuffle

trait ShuffleReader[K, C] {
  def read(): Iterator[Product2[K, C]]
}
Note
ShuffleReader is a private[spark] contract.
Table 1. ShuffleReader Contract
Method Description

read

Reading combined key-value records for a reduce task

Used when:

  • CoGroupedRDD, ShuffledRDD, and SubtractedRDD are requested to compute a partition (for a ShuffleDependency dependency)

  • Spark SQL’s ShuffledRowRDD is requested to compute a partition

Note
BlockStoreShuffleReader is the one and only known ShuffleReader in Apache Spark.

results matching ""

    No results matching ""