StreamingQueryListenerBus — Event Bus for Streaming Events

StreamingQueryListenerBus is an event bus (ListenerBus[StreamingQueryListener, StreamingQueryListener.Event]) for dispatching streaming life-cycle events of active streaming queries (that eventually are delivered to StreamingQueryListeners).

StreamingQueryListenerBus is created for StreamingQueryManager (once per SparkSession).

Figure 1. StreamingQueryListenerBus is Created Once In SparkSession

StreamingQueryListenerBus is also a SparkListener and registers itself with the LiveListenerBus (of the SparkSession) to intercept QueryStartedEvents.

Creating StreamingQueryListenerBus Instance

StreamingQueryListenerBus takes the following when created:

LiveListenerBus

StreamingQueryListenerBus registers itself with the LiveListenerBus.

Run IDs of Active Streaming Queries

activeQueryRunIds: HashSet[UUID]

activeQueryRunIds is an internal registry of run IDs of active streaming queries in the SparkSession.

A runId is added when StreamingQueryListenerBus is requested to post a QueryStartedEvent
A runId is removed when StreamingQueryListenerBus is requested to post a QueryTerminatedEvent

activeQueryRunIds is used internally to dispatch a streaming event to a StreamingQueryListener (so the events gets sent out to streaming queries in the SparkSession).

Posting Streaming Event to LiveListenerBus — `post` Method

post(event: StreamingQueryListener.Event): Unit

post simply posts the input event directly to the LiveListenerBus unless it is a QueryStartedEvent.

For a QueryStartedEvent, post adds the runId (of the streaming query that has been started) to the activeQueryRunIds internal registry first, posts the event to the LiveListenerBus and then postToAll.

Note	`post` is used exclusively when `StreamingQueryManager` is requested to post a streaming event.

`doPostEvent` Method

doPostEvent(
  listener: StreamingQueryListener,
  event: StreamingQueryListener.Event): Unit

Note	`doPostEvent` is part of Spark Core’s `ListenerBus` contract to post an event to the specified listener.

doPostEvent branches per the type of StreamingQueryListener.Event:

For a QueryStartedEvent, requests the StreamingQueryListener to onQueryStarted
For a QueryProgressEvent, requests the StreamingQueryListener to onQueryProgress
For a QueryTerminatedEvent, requests the StreamingQueryListener to onQueryTerminated

For any other event, doPostEvent simply does nothing (swallows it).

`postToAll` Method

postToAll(event: Event): Unit

Note	`postToAll` is part of Spark Core’s `ListenerBus` contract to post an event to all registered listeners.

postToAll first requests the parent ListenerBus to post the event to all registered listeners.

For a QueryTerminatedEvent, postToAll simply removes the runId (of the streaming query that has been terminated) from the activeQueryRunIds internal registry.

StreamingQueryListenerBus