StreamingQuery

StreamingQuery is the contract for a streaming query that is executed continuously and concurrently (i.e. on a separate thread).

Note
StreamingQuery is called continuous query or streaming query.
Note
StreamingQuery is a Scala trait with the only implementation being StreamExecution (and less importanly StreamingQueryWrapper for serializing a non-serializable StreamExecution).

StreamingQuery can be in two states:

  • active (started)

  • inactive (stopped)

If inactive, StreamingQuery may have transitioned into the state due to an StreamingQueryException (that is available under exception).

StreamingQuery tracks current state of all the sources, i.e. SourceStatus, as sourceStatuses.

There could only be a single Sink for a StreamingQuery with many Sources.

StreamingQuery can be stopped by stop or an exception.

Table 1. StreamingQuery Contract
Method Description

awaitTermination

awaitTermination(): Unit
awaitTermination(timeoutMs: Long): Boolean

Used when…​FIXME

exception

exception: Option[StreamingQueryException]

StreamingQueryException if the query has finished due to an exception

Used when…​FIXME

explain

explain(): Unit
explain(extended: Boolean): Unit

Used when…​FIXME

id

id: UUID

The unique identifier of the streaming query

Used when…​FIXME

isActive

isActive: Boolean

Used when…​FIXME

lastProgress

lastProgress: StreamingQueryProgress

The last StreamingQueryProgress of the streaming query

Used when…​FIXME

name

name: String

The name of the query that is unique across all active queries

Used when…​FIXME

processAllAvailable

processAllAvailable(): Unit

Waits the streaming query until there are no data available in sources or the query has been terminated.

Used when…​FIXME

recentProgress

recentProgress: Array[StreamingQueryProgress]

Collection of the recent StreamingQueryProgress updates.

Used when…​FIXME

runId

runId: UUID

The unique identifier of the current execution of the streaming query

Used when…​FIXME

sparkSession

sparkSession: SparkSession

Used when…​FIXME

status

status: StreamingQueryStatus

The StreamingQueryStatus of the streaming query (as StreamExecution has accumulated being a ProgressReporter while running the streaming query)

Used when…​FIXME

stop

stop(): Unit

Stops the streaming query

Used exclusively when StreamingQueryWrapper is requested to stop

results matching ""

    No results matching ""