StreamingQueryListener — Intercepting Life Cycle Events of Streaming Queries

StreamingQueryListener is the contract of listeners that want to be notified about the life cycle events of streaming queries, i.e. start, progress and termination.

Table 1. StreamingQueryListener Contract
Method Description

onQueryStarted

onQueryStarted(
  event: QueryStartedEvent): Unit

Informs that DataStreamWriter was requested to start execution of the streaming query (on the stream execution thread)

onQueryProgress

onQueryProgress(
  event: QueryProgressEvent): Unit

Informs that MicroBatchExecution has finished triggerExecution phase (the end of a streaming batch)

onQueryTerminated

onQueryTerminated(
  event: QueryTerminatedEvent): Unit

Informs that a streaming query was stopped or terminated due to an error

StreamingQueryListener is informed about the life cycle events when StreamingQueryListenerBus is requested to doPostEvent.

Table 2. StreamingQueryListener’s Life Cycle Events and Callbacks
Event Callback Description

QueryStartedEvent

onQueryStarted

Posted when StreamExecution is requested to run stream processing (when DataStreamWriter is requested to start execution of the streaming query on the stream execution thread)

QueryProgressEvent

onQueryProgress

Posted when ProgressReporter is requested to update progress of a streaming query (after MicroBatchExecution has finished triggerExecution phase at the end of a streaming batch)

QueryTerminatedEvent

onQueryTerminated

Posted when StreamExecution is requested to run stream processing (and the streaming query was stopped or terminated due to an error)

You can register a StreamingQueryListener using StreamingQueryManager.addListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.addListener(queryListener)

You can remove a StreamingQueryListener using StreamingQueryManager.removeListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.removeListener(queryListener)
StreamingQueryListener onQueryStarted.png
Figure 1. StreamingQueryListener Notified about Query’s Start (onQueryStarted)
Note
onQueryStarted is used internally to unblock the starting thread of StreamExecution.
StreamingQueryListener onQueryProgress.png
Figure 2. StreamingQueryListener Notified about Query’s Progress (onQueryProgress)
StreamingQueryListener onQueryTerminated.png
Figure 3. StreamingQueryListener Notified about Query’s Termination (onQueryTerminated)
Note

You can also register a streaming event listener using the general SparkListener interface.

results matching ""

    No results matching ""