StreamingQuery Contract

StreamingQuery is the contract of streaming queries that are executed continuously and concurrently (i.e. on a separate thread).

Note
StreamingQuery is called continuous query or streaming query.
Note
StreamingQuery is a Scala trait with the only implementation being StreamExecution (and less importanly StreamingQueryWrapper for serializing a non-serializable StreamExecution).
Table 1. StreamingQuery Contract
Method Description

awaitTermination

awaitTermination(): Unit
awaitTermination(timeoutMs: Long): Boolean

Used when…​FIXME

exception

exception: Option[StreamingQueryException]

StreamingQueryException if the query has finished due to an exception

Used when…​FIXME

explain

explain(): Unit
explain(extended: Boolean): Unit

Used when…​FIXME

id

id: UUID

The unique identifier of the streaming query (that does not change across restarts unlike runId)

Used when…​FIXME

isActive

isActive: Boolean

Indicates whether the streaming query is active (true) or not (false)

Used when…​FIXME

lastProgress

lastProgress: StreamingQueryProgress

The last StreamingQueryProgress of the streaming query

Used when…​FIXME

name

name: String

The name of the query that is unique across all active queries

Used when…​FIXME

processAllAvailable

processAllAvailable(): Unit

Pauses (blocks) the current thread until the streaming query has no more data to be processed or has been stopped

Intended for testing

recentProgress

recentProgress: Array[StreamingQueryProgress]

Collection of the recent StreamingQueryProgress updates.

Used when…​FIXME

runId

runId: UUID

The unique identifier of the current execution of the streaming query (that is different every restart unlike id)

Used when…​FIXME

sparkSession

sparkSession: SparkSession

Used when…​FIXME

status

status: StreamingQueryStatus

StreamingQueryStatus of the streaming query (as StreamExecution has accumulated being a ProgressReporter while running the streaming query)

Used when…​FIXME

stop

stop(): Unit

Stops the streaming query

StreamingQuery can be in two states:

  • active (started)

  • inactive (stopped)

If inactive, StreamingQuery may have transitioned into the state due to an StreamingQueryException (that is available under exception).

StreamingQuery tracks current state of all the sources, i.e. SourceStatus, as sourceStatuses.

There could only be a single Sink for a StreamingQuery with many Sources.

StreamingQuery can be stopped by stop or an exception.

results matching ""

    No results matching ""