StorageStatusListener Spark Listener

StorageStatusListener is a SparkListener that tracks status of BlockManagers on "nodes" in a Spark application, i.e. the driver and the executors.

StorageStatusListener is created and registered when a SparkUI is created. It is later used to create ExecutorsListener and StorageListener Spark listeners.
Table 1. StorageStatusListener Registries
Registry Description


The lookup table for StorageStatus per executor or the driver.


A collection of StorageStatuss of removed/inactive BlockManagers.

Table 2. StorageStatusListener Event Handlers
Event Handler Description


Removes the RDD blocks for the unpersisted rddId from every StorageStatuss in executorIdToStorageStatus internal registry.


Registers a BlockManager on an executor in executorIdToStorageStatus internal registry.

Removes any other BlockManager that may have been registered for the executor earlier in deadExecutorStorageStatus internal registry.


Removes an executor from executorIdToStorageStatus internal registry and adds the removed StorageStatus to deadExecutorStorageStatus internal registry.

Removes the oldest StorageStatus when the number of entries in deadExecutorStorageStatus is bigger than spark.ui.retainedDeadExecutors.


Updates StorageStatus for an executor in executorIdToStorageStatus internal registry, i.e. removes a block for NONE storage level and updates otherwise.

results matching ""

    No results matching ""