ExecutionListenerManager — Management Interface of QueryExecutionListeners

ExecutionListenerManager is the management interface for QueryExecutionListeners that listen for execution metrics:

Name of the action (that triggered a query execution)
QueryExecution
Execution time of this query (in nanoseconds)

ExecutionListenerManager is available as listenerManager property of SparkSession (and listenerManager property of SessionState).

scala> :type spark.listenerManager
org.apache.spark.sql.util.ExecutionListenerManager

scala> :type spark.sessionState.listenerManager
org.apache.spark.sql.util.ExecutionListenerManager

ExecutionListenerManager takes a single SparkConf when created

While created, ExecutionListenerManager reads spark.sql.queryExecutionListeners configuration property with QueryExecutionListeners and registers them.

ExecutionListenerManager uses spark.sql.queryExecutionListeners configuration property as the list of QueryExecutionListeners that should be automatically added to newly created sessions (and registers them while being created).

Table 1. ExecutionListenerManager’s Public Methods
Method	Description
register	`register(listener: QueryExecutionListener): Unit`
unregister	`unregister(listener: QueryExecutionListener): Unit`
clear	`clear(): Unit`

ExecutionListenerManager is created exclusively when BaseSessionStateBuilder is requested for ExecutionListenerManager (while SessionState is built).

ExecutionListenerManager uses listeners internal registry for registered QueryExecutionListeners.

`onSuccess` Internal Method

onSuccess(funcName: String, qe: QueryExecution, duration: Long): Unit

onSuccess…FIXME

Note	`onSuccess` is used when: `DataFrameWriter` is requested to run a logical command (after it has finished with no exceptions) `Dataset` is requested to withAction

`onFailure` Internal Method

onFailure(funcName: String, qe: QueryExecution, exception: Exception): Unit

onFailure…FIXME

Note	`onFailure` is used when: `DataFrameWriter` is requested to run a logical command (after it has reported an exception) `Dataset` is requested to withAction

`withErrorHandling` Internal Method

withErrorHandling(f: QueryExecutionListener => Unit): Unit

withErrorHandling…FIXME

Note	`withErrorHandling` is used when `ExecutionListenerManager` is requested to onSuccess and onFailure.

Registering QueryExecutionListener — `register` Method

register(listener: QueryExecutionListener): Unit

Internally, register simply registers (adds) the input QueryExecutionListener to the listeners internal registry.

ExecutionListenerManager — Management Interface of QueryExecutionListeners