StreamingQueryListenerBus — Event Bus for Streaming Events

StreamingQueryListenerBus is an event bus (ListenerBus[StreamingQueryListener, StreamingQueryListener.Event]) for dispatching streaming life-cycle events of active streaming queries (that eventually are delivered to StreamingQueryListeners).

StreamingQueryListenerBus is created for StreamingQueryManager (once per SparkSession).

Figure 1. StreamingQueryListenerBus is Created Once In SparkSession

StreamingQueryListenerBus is also a SparkListener and registers itself with the LiveListenerBus (of the SparkSession) to intercept QueryStartedEvents.

Creating StreamingQueryListenerBus Instance

StreamingQueryListenerBus takes the following when created:

  • LiveListenerBus

StreamingQueryListenerBus registers itself with the LiveListenerBus.

Run IDs of Active Streaming Queries

activeQueryRunIds: HashSet[UUID]

activeQueryRunIds is an internal registry of run IDs of active streaming queries in the SparkSession.

activeQueryRunIds is used internally to dispatch a streaming event to a StreamingQueryListener (so the events gets sent out to streaming queries in the SparkSession).

Posting Streaming Event to LiveListenerBus — post Method

post(event: StreamingQueryListener.Event): Unit

post simply posts the input event directly to the LiveListenerBus unless it is a QueryStartedEvent.

For a QueryStartedEvent, post adds the runId (of the streaming query that has been started) to the activeQueryRunIds internal registry first, posts the event to the LiveListenerBus and then postToAll.

post is used exclusively when StreamingQueryManager is requested to post a streaming event.

doPostEvent Method

  listener: StreamingQueryListener,
  event: StreamingQueryListener.Event): Unit
doPostEvent is part of Spark Core’s ListenerBus contract to post an event to the specified listener.

doPostEvent branches per the type of StreamingQueryListener.Event:

For any other event, doPostEvent simply does nothing (swallows it).

postToAll Method

postToAll(event: Event): Unit
postToAll is part of Spark Core’s ListenerBus contract to post an event to all registered listeners.

postToAll first requests the parent ListenerBus to post the event to all registered listeners.

For a QueryTerminatedEvent, postToAll simply removes the runId (of the streaming query that has been terminated) from the activeQueryRunIds internal registry.

results matching ""

    No results matching ""