SpillListener — Detecting Spills in Jobs (for Testing)

SpillListener is a SparkListener that intercepts (listens to) the following events:

SpillListener is created exclusively for TestUtils when requested for assertSpilled and assertNotSpilled.

SpillListener takes no input arguments to be created.

Note
SpillListener is a Scala private class.
Table 1. SpillListener’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description

spilledStageIds

(mutable.HashMap[Int, ArrayBuffer[TaskMetrics]])

Used when…​FIXME

stageIdToTaskMetrics

(mutable.HashSet[Int])

Used when…​FIXME

onTaskEnd Callback

onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit
Note
onTaskEnd is part of the SparkListener Contract to…​FIXME.

onTaskEnd…​FIXME

onStageCompleted Callback

onStageCompleted(stageComplete: SparkListenerStageCompleted): Unit
Note
onStageCompleted is part of the SparkListener Contract to…​FIXME.

onStageCompleted…​FIXME

results matching ""

    No results matching ""