AppStatusListener

AppStatusListener is a SparkListener that AppStatusStore uses to…​FIXME

AppStatusListener is created when:

Table 1. AppStatusListener’s Event Handlers
Event Handler

SparkListenerApplicationStart

onApplicationStart

SparkListenerApplicationEnd

onApplicationEnd

SparkListenerBlockManagerAdded

onBlockManagerAdded

SparkListenerBlockManagerRemoved

onBlockManagerRemoved

SparkListenerBlockUpdated

onBlockUpdated

SparkListenerEnvironmentUpdate

onEnvironmentUpdate

SparkListenerEvent

onOtherEvent

SparkListenerExecutorAdded

onExecutorAdded

SparkListenerExecutorBlacklisted

onExecutorBlacklisted

SparkListenerExecutorMetricsUpdate

onExecutorMetricsUpdate

SparkListenerExecutorRemoved

onExecutorRemoved

SparkListenerExecutorUnblacklisted

onExecutorUnblacklisted

SparkListenerJobStart

onJobStart

SparkListenerJobEnd

onJobEnd

SparkListenerNodeBlacklisted

onNodeBlacklisted

SparkListenerNodeUnblacklisted

onNodeUnblacklisted

SparkListenerStageCompleted

onStageCompleted

SparkListenerStageSubmitted

onStageSubmitted

SparkListenerTaskEnd

onTaskEnd

SparkListenerTaskGettingResult

onTaskGettingResult

SparkListenerTaskStart

onTaskStart

SparkListenerUnpersistRDD

onUnpersistRDD

Table 2. AppStatusListener’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description

appInfo

v1.ApplicationInfo

appSummary

AppSummary

liveUpdatePeriodNs

coresPerTask

Default: 1

liveRDDs

LiveRDDs by RDD ID

liveStages

LiveStages by (Int, Int)

liveTasks

LiveTask by task ID

liveJobs

LiveJob by job ID

liveExecutors

LiveExecutor by executor ID

pools

SchedulerPool by FIXME

activeExecutorCount

Number of active executors

onStageSubmitted Method

onStageSubmitted(event: SparkListenerStageSubmitted): Unit
Note
onStageSubmitted is part of SparkListener Contract to…​FIXME.

onStageSubmitted…​FIXME

update Internal Method

update(entity: LiveEntity, now: Long, last: Boolean = false): Unit

update simply requests the LiveEntity to write (with the ElementTrackingStore as the store and the last flag as checkTriggers flag).

Note
update is used in event handlers (i.e. onApplicationStart, onExecutorRemoved, onJobEnd, onStageSubmitted, onTaskEnd, onStageCompleted), liveUpdate, maybeUpdate, flush and updateRDDBlock.

flush Internal Method

flush(): Unit

flush…​FIXME

Note
flush is used when…​FIXME

maybeUpdate Internal Method

maybeUpdate(entity: LiveEntity, now: Long): Unit

maybeUpdate…​FIXME

Note
maybeUpdate is used when…​FIXME

liveUpdate Internal Method

liveUpdate(entity: LiveEntity, now: Long): Unit

liveUpdate…​FIXME

Note
liveUpdate is used when…​FIXME

updateStreamBlock Internal Method

updateStreamBlock(event: SparkListenerBlockUpdated, stream: StreamBlockId): Unit

updateStreamBlock…​FIXME

Note
updateStreamBlock is used exclusively when AppStatusListener is requested to handle a SparkListenerBlockUpdated event (for a StreamBlockId).

Intercepting SparkListenerBlockUpdated Events — onBlockUpdated Handler Method

onBlockUpdated(event: SparkListenerBlockUpdated): Unit
Note
onBlockUpdated is part of SparkListener Contract to…​FIXME.

onBlockUpdated simply dispatches to the following event-specific handlers (per BlockId type):

Creating AppStatusListener Instance

AppStatusListener takes the following when created:

AppStatusListener initializes the internal registries and counters.

updateRDDBlock Internal Method

updateRDDBlock(event: SparkListenerBlockUpdated, block: RDDBlockId): Unit

updateRDDBlock…​FIXME

Note
updateRDDBlock is used exclusively when AppStatusListener is requested to handle a SparkListenerBlockUpdated event (for a RDDBlockId).

results matching ""

    No results matching ""