StreamingQueryListener — Intercepting Streaming Events

StreamingQueryListener is the contract for listeners that want to be notified about the life cycle events of streaming queries, i.e. start, progress and termination of a query.

package org.apache.spark.sql.streaming

abstract class StreamingQueryListener {
  def onQueryStarted(event: QueryStartedEvent): Unit
  def onQueryProgress(event: QueryProgressEvent): Unit
  def onQueryTerminated(event: QueryTerminatedEvent): Unit
Table 1. StreamingQueryListener’s Life Cycle Events and Callbacks
Event Callback When Posted


  • id

  • runId

  • name


Right after StreamExecution has started running streaming batches.



ProgressReporter reports query progress (which is when StreamExecution runs batches and a trigger has finished).


  • id

  • runId

  • Optional exception if terminated due to an error


Right before StreamExecution finishes running streaming batches (due to a stop or an exception).

You can register a StreamingQueryListener using StreamingQueryManager.addListener method.

val queryListener: StreamingQueryListener = ...

You can remove a StreamingQueryListener using StreamingQueryManager.removeListener method.

val queryListener: StreamingQueryListener = ...
StreamingQueryListener onQueryStarted.png
Figure 1. StreamingQueryListener Notified about Query’s Start (onQueryStarted)
onQueryStarted is used internally to unblock the starting thread of StreamExecution.
StreamingQueryListener onQueryProgress.png
Figure 2. StreamingQueryListener Notified about Query’s Progress (onQueryProgress)
StreamingQueryListener onQueryTerminated.png
Figure 3. StreamingQueryListener Notified about Query’s Termination (onQueryTerminated)

You can also register a streaming event listener using the general SparkListener interface.

Details on SparkListener interface can be found in the Mastering Apache Spark 2 gitbook.

results matching ""

    No results matching ""