StreamingQuery Contract

StreamingQuery is the contract of streaming queries that are executed continuously and concurrently (i.e. on a separate thread).

Note
 StreamingQuery is called continuous query or streaming query.
Note
 StreamingQuery is a Scala trait with the only implementation being StreamExecution (and less importanly StreamingQueryWrapper for serializing a non-serializable StreamExecution).
Table 1. StreamingQuery Contract
Method Description

awaitTermination

 
awaitTermination(): Unit
awaitTermination(timeoutMs: Long): Boolean

Used when…​FIXME

exception

 
exception: Option[StreamingQueryException]

StreamingQueryException if the query has finished due to an exception

Used when…​FIXME

explain

 
explain(): Unit
explain(extended: Boolean): Unit

Used when…​FIXME

id

 
id: UUID

The unique identifier of the streaming query (that does not change across restarts unlike runId)

Used when…​FIXME

isActive

 
isActive: Boolean

Indicates whether the streaming query is active (true) or not (false)

Used when…​FIXME

lastProgress

 
lastProgress: StreamingQueryProgress

The last StreamingQueryProgress of the streaming query

Used when…​FIXME

name

 
name: String

The name of the query that is unique across all active queries

Used when…​FIXME

processAllAvailable

 
processAllAvailable(): Unit

Pauses (blocks) the current thread until the streaming query has no more data to be processed or has been stopped

Intended for testing

recentProgress

 
recentProgress: Array[StreamingQueryProgress]

Collection of the recent StreamingQueryProgress updates.

Used when…​FIXME

runId

 
runId: UUID

The unique identifier of the current execution of the streaming query (that is different every restart unlike id)

Used when…​FIXME

sparkSession

 
sparkSession: SparkSession

Used when…​FIXME

status

 
status: StreamingQueryStatus

StreamingQueryStatus of the streaming query (as StreamExecution has accumulated being a ProgressReporter while running the streaming query)

Used when…​FIXME

stop

 
stop(): Unit

Stops the streaming query

StreamingQuery can be in two states:

  • active (started)

  • inactive (stopped)

If inactive, StreamingQuery may have transitioned into the state due to an StreamingQueryException (that is available under exception).

StreamingQuery tracks current state of all the sources, i.e. SourceStatus, as sourceStatuses.

There could only be a single Sink for a StreamingQuery with many Sources.

StreamingQuery can be stopped by stop or an exception.

results matching ""

    No results matching ""