ShuffleWriter Contract

ShuffleWriter is the contract of shuffle writers that can write a sequence of key-value records (of a RDD partition) to a shuffle system.

ShuffleWriter is used exclusively when ShuffleMapTask is requested to run.

Table 1. ShuffleWriter Contract
Method Description

stop

stop(success: Boolean): Option[MapStatus]

Stops (closes) the ShuffleWriter and returns a MapStatus if the writing completed successfully. The success flag is the status of the task execution.

Used exclusively when ShuffleMapTask is requested to run

write

write(records: Iterator[Product2[K, V]]): Unit

Writes a sequence of key-value records to a shuffle system

Used exclusively when ShuffleMapTask is requested to run

Table 2. ShuffleWriters
ShuffleWriter Description

BypassMergeSortShuffleWriter

ShuffleWriter for BypassMergeSortShuffleHandles

SortShuffleWriter

Fallback ShuffleWriter (when neither BypassMergeSortShuffleWriter nor UnsafeShuffleWriter could be used)

UnsafeShuffleWriter

ShuffleWriter for SerializedShuffleHandles

results matching ""

    No results matching ""