ExchangeCoordinator

ExchangeCoordinator is created when EnsureRequirements physical query optimization is requested to add an ExchangeCoordinator for Adaptive Query Execution.

ExchangeCoordinator takes the following to be created:

Number of ShuffleExchangeExec unary physical operators
Recommended size of the input data of a post-shuffle partition (configured by spark.sql.adaptive.shuffle.targetPostShuffleInputSize property)
Optional advisory minimum number of post-shuffle partitions (default: None) (configured by spark.sql.adaptive.minNumPostShufflePartitions property)

ExchangeCoordinator keeps track of ShuffleExchangeExec unary physical operators that were registered (when ShuffleExchangeExec unary physical operator was requested to prepare itself for execution).

ExchangeCoordinator uses the following text representation (i.e. toString):

coordinator[target post-shuffle partition size: [advisoryTargetPostShuffleInputSize]]

postShuffleRDD(exchange: ShuffleExchangeExec): ShuffledRowRDD

postShuffleRDD…FIXME

Note	`postShuffleRDD` is used exclusively when `ShuffleExchangeExec` unary physical operator is requested to execute.

doEstimationIfNecessary(): Unit

doEstimationIfNecessary…FIXME

Note	`doEstimationIfNecessary` is used exclusively when `ExchangeCoordinator` is requested for a post-shuffle RDD (ShuffledRowRDD).

estimatePartitionStartIndices(
  mapOutputStatistics: Array[MapOutputStatistics]): Array[Int]

estimatePartitionStartIndices…FIXME

Note	`estimatePartitionStartIndices` is used exclusively when `ExchangeCoordinator` is requested for a doEstimationIfNecessary.

registerExchange(exchange: ShuffleExchangeExec): Unit

registerExchange simply adds the ShuffleExchangeExec unary physical operator to the exchanges internal registry.

Note	`registerExchange` is used exclusively when `ShuffleExchangeExec` unary physical operator is requested to prepare itself for execution.

results matching ""