GroupStateTimeout

GroupStateTimeout represents the possible timeouts that you can use for the state-aware Dataset operations:

GroupStateTimeout is part of org.apache.spark.sql.streaming package.

import org.apache.spark.sql.streaming.GroupStateTimeout
Table 1. Types of GroupStateTimeouts (in alphabetical order)
GroupStateTimeout Description

ProcessingTimeTimeout

Timeout based on the processing time.

Note

FlatMapGroupsWithStateExec requires that batchTimestampMs is specified when ProcessingTimeTimeout is used.

batchTimestampMs is defined when IncrementalExecution is created (and so is state). IncrementalExecution is given OffsetSeqMetadata when StreamExecution runs a streaming batch.

Caution
FIXME Describe OffsetSeqMetadata and StreamExecution.offsetSeqMetadata

EventTimeTimeout

Timeout based on the event time

Used when…​FIXME

NoTimeout

No timeout

Used when…​FIXME

results matching ""

    No results matching ""