StreamingQueryListener — Intercepting Life Cycle Events of Streaming Queries

StreamingQueryListener is the contract of listeners that want to be notified about the life cycle events of streaming queries, i.e. start, progress and termination.

Table 1. StreamingQueryListener Contract
Method	Description
`onQueryStarted`	`onQueryStarted( event: QueryStartedEvent): Unit` Informs that `DataStreamWriter` was requested to start execution of the streaming query (on the stream execution thread)
`onQueryProgress`	`onQueryProgress( event: QueryProgressEvent): Unit` Informs that `MicroBatchExecution` has finished triggerExecution phase (the end of a streaming batch)
`onQueryTerminated`	`onQueryTerminated( event: QueryTerminatedEvent): Unit` Informs that a streaming query was stopped or terminated due to an error

StreamingQueryListener is informed about the life cycle events when StreamingQueryListenerBus is requested to doPostEvent.

Table 2. StreamingQueryListener’s Life Cycle Events and Callbacks
Event	Callback	Description
QueryStartedEvent id runId name	onQueryStarted	Posted when `StreamExecution` is requested to run stream processing (when `DataStreamWriter` is requested to start execution of the streaming query on the stream execution thread)
QueryProgressEvent StreamingQueryProgress	onQueryProgress	Posted when `ProgressReporter` is requested to update progress of a streaming query (after `MicroBatchExecution` has finished triggerExecution phase at the end of a streaming batch)
QueryTerminatedEvent id runId exception if terminated due to an error	onQueryTerminated	Posted when `StreamExecution` is requested to run stream processing (and the streaming query was stopped or terminated due to an error)

You can register a StreamingQueryListener using StreamingQueryManager.addListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.addListener(queryListener)

You can remove a StreamingQueryListener using StreamingQueryManager.removeListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.removeListener(queryListener)

Figure 1. StreamingQueryListener Notified about Query’s Start (onQueryStarted)

Note	`onQueryStarted` is used internally to unblock the starting thread of `StreamExecution`.

Figure 2. StreamingQueryListener Notified about Query’s Progress (onQueryProgress)

Figure 3. StreamingQueryListener Notified about Query’s Termination (onQueryTerminated)

Note	You can also register a streaming event listener using the general `SparkListener` interface. Read up on SparkListener in the The Internals of Apache Spark book.

StreamingQueryListener