FlatMapGroupsWithState Unary Logical Operator

FlatMapGroupsWithState is a unary logical operator (i.e. LogicalPlan) that is created to represent the following operators in KeyValueGroupedDataset:

Note

FlatMapGroupsWithState is translated to:

  • FlatMapGroupsWithStateExec physical operator in FlatMapGroupsWithStateStrategy execution planning strategy for streaming Datasets (aka streaming plans)

  • MapGroupsExec physical operator in BasicOperators execution planning strategy for non-streaming/batch Datasets (aka batch plans)

Creating SerializeFromObject with FlatMapGroupsWithState — apply Factory Method

apply[K: Encoder, V: Encoder, S: Encoder, U: Encoder](
  func: (Any, Iterator[Any], LogicalGroupState[Any]) => Iterator[Any],
  groupingAttributes: Seq[Attribute],
  dataAttributes: Seq[Attribute],
  outputMode: OutputMode,
  isMapGroupsWithState: Boolean,
  timeout: GroupStateTimeout,
  child: LogicalPlan): LogicalPlan

apply creates a SerializeFromObject logical operator with a FlatMapGroupsWithState as its child logical operator.

Internally, apply creates SerializeFromObject object consumer (aka unary logical operator) with FlatMapGroupsWithState logical plan.

Internally, apply finds ExpressionEncoder for the type S and creates a FlatMapGroupsWithState with UnresolvedDeserializer for the types K and V.

In the end, apply creates a SerializeFromObject object consumer with the FlatMapGroupsWithState.

Note
apply is used when flatMapGroupsWithState is executed.

Creating FlatMapGroupsWithState Instance

FlatMapGroupsWithState takes the following when created:

  • State function of type (Any, Iterator[Any], LogicalGroupState[Any]) ⇒ Iterator[Any]

  • Key deserializer Catalyst expression

  • Value deserializer Catalyst expression

  • Grouping attributes

  • Data attributes

  • Output object attribute

  • State ExpressionEncoder

  • Output mode

  • isMapGroupsWithState Flag (disabled by default)

  • GroupStateTimeout

  • Child logical operator

results matching ""

    No results matching ""