StreamingAggregationStateManagerBaseImpl — Base State Manager for Streaming Aggregation

StreamingAggregationStateManagerBaseImpl is the base implementation of the StreamingAggregationStateManager contract for state managers for streaming aggregations that use UnsafeProjection to getKey.

StreamingAggregationStateManagerBaseImpl uses UnsafeProjection to getKey.

Table 1. StreamingAggregationStateManagerBaseImpls
StreamingAggregationStateManagerBaseImpl Description

StreamingAggregationStateManagerImplV1

Legacy StreamingAggregationStateManager (used when spark.sql.streaming.aggregation.stateFormatVersion configuration property is 1)

StreamingAggregationStateManagerImplV2

Default StreamingAggregationStateManager (used when spark.sql.streaming.aggregation.stateFormatVersion configuration property is 2)

StreamingAggregationStateManagerBaseImpl takes the following to be created:

  • Catalyst expressions for the keys (Seq[Attribute])

  • Catalyst expressions for the input rows (Seq[Attribute])

Note
StreamingAggregationStateManagerBaseImpl is a Scala abstract class and cannot be created directly. It is created indirectly for the concrete StreamingAggregationStateManagerBaseImpls.

Committing (Changes to) State Store — commit Method

commit(store: StateStore): Long
Note
commit is part of the StreamingAggregationStateManager Contract to commit changes to a state store.

commit simply requests the state store to commit state changes.

Removing Key From State Store — remove Method

remove(store: StateStore, key: UnsafeRow): Unit
Note
remove is part of the StreamingAggregationStateManager Contract to remove a key from a state store.

remove…​FIXME

getKey Method

getKey(row: UnsafeRow): UnsafeRow
Note
getKey is part of the StreamingAggregationStateManager Contract to…​FIXME

getKey…​FIXME

Getting All Keys in State Store — keys Method

keys(store: StateStore): Iterator[UnsafeRow]
Note
keys is part of the StreamingAggregationStateManager Contract to get all keys in a state store (as an iterator).

keys…​FIXME

results matching ""

    No results matching ""