StreamingQueryListener — Intercepting Streaming Events

StreamingQueryListener is the contract for listeners that want to be notified about the life cycle events of streaming queries, i.e. start, progress and termination of a query.

package org.apache.spark.sql.streaming

abstract class StreamingQueryListener {
  def onQueryStarted(event: QueryStartedEvent): Unit
  def onQueryProgress(event: QueryProgressEvent): Unit
  def onQueryTerminated(event: QueryTerminatedEvent): Unit
}
Table 1. StreamingQueryListener’s Life Cycle Events and Callbacks
Event Callback When Posted

QueryStartedEvent

  • id

  • runId

  • name

onQueryStarted

Right after StreamExecution has started running streaming batches.

QueryProgressEvent

onQueryProgress

ProgressReporter reports query progress (which is when StreamExecution runs batches and a trigger has finished).

QueryTerminatedEvent

  • id

  • runId

  • Optional exception if terminated due to an error

onQueryTerminated

Right before StreamExecution finishes running streaming batches (due to a stop or an exception).

You can register a StreamingQueryListener using StreamingQueryManager.addListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.addListener(queryListener)

You can remove a StreamingQueryListener using StreamingQueryManager.removeListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.removeListener(queryListener)
StreamingQueryListener onQueryStarted.png
Figure 1. StreamingQueryListener Notified about Query’s Start (onQueryStarted)
Note
onQueryStarted is used internally to unblock the starting thread of StreamExecution.
StreamingQueryListener onQueryProgress.png
Figure 2. StreamingQueryListener Notified about Query’s Progress (onQueryProgress)
StreamingQueryListener onQueryTerminated.png
Figure 3. StreamingQueryListener Notified about Query’s Termination (onQueryTerminated)
Note

You can also register a streaming event listener using the general SparkListener interface.

Details on SparkListener interface can be found in the Mastering Apache Spark 2 gitbook.

results matching ""

    No results matching ""