StreamingQuery Contract

StreamingQuery is the contract of streaming queries that are executed continuously and concurrently (i.e. on a separate thread).

StreamingQuery is called continuous query or streaming query.
StreamingQuery is a Scala trait with the only implementation being StreamExecution (and less importanly StreamingQueryWrapper for serializing a non-serializable StreamExecution).
Table 1. StreamingQuery Contract
Method Description


awaitTermination(): Unit
awaitTermination(timeoutMs: Long): Boolean

Used when…​FIXME


exception: Option[StreamingQueryException]

StreamingQueryException if the query has finished due to an exception

Used when…​FIXME


explain(): Unit
explain(extended: Boolean): Unit

Used when…​FIXME


id: UUID

The unique identifier of the streaming query (that does not change across restarts unlike runId)

Used when…​FIXME


isActive: Boolean

Indicates whether the streaming query is active (true) or not (false)

Used when…​FIXME


lastProgress: StreamingQueryProgress

The last StreamingQueryProgress of the streaming query

Used when…​FIXME


name: String

The name of the query that is unique across all active queries

Used when…​FIXME


processAllAvailable(): Unit

Pauses (blocks) the current thread until the streaming query has no more data to be processed or has been stopped

Intended for testing


recentProgress: Array[StreamingQueryProgress]

Collection of the recent StreamingQueryProgress updates.

Used when…​FIXME


runId: UUID

The unique identifier of the current execution of the streaming query (that is different every restart unlike id)

Used when…​FIXME


sparkSession: SparkSession

Used when…​FIXME


status: StreamingQueryStatus

StreamingQueryStatus of the streaming query (as StreamExecution has accumulated being a ProgressReporter while running the streaming query)

Used when…​FIXME


stop(): Unit

Stops the streaming query

StreamingQuery can be in two states:

  • active (started)

  • inactive (stopped)

If inactive, StreamingQuery may have transitioned into the state due to an StreamingQueryException (that is available under exception).

StreamingQuery tracks current state of all the sources, i.e. SourceStatus, as sourceStatuses.

There could only be a single Sink for a StreamingQuery with many Sources.

StreamingQuery can be stopped by stop or an exception.

results matching ""

    No results matching ""