StreamingGlobalLimitExec Unary Physical Operator

StreamingGlobalLimitExec is a unary physical operator that represents a Limit logical operator of a streaming query at execution time.

Note

A unary physical operator (UnaryExecNode) is a physical operator with a single child physical operator.

Read up on UnaryExecNode (and physical operators in general) in The Internals of Spark SQL book.

StreamingGlobalLimitExec is created exclusively when StreamingGlobalLimitStrategy execution planning strategy is requested to plan a Limit logical operator (in the logical plan of a streaming query) for execution.

Note

Limit logical operator represents Dataset.limit operator in a logical query plan.

StreamingGlobalLimitExec supports Append output mode only.

The optional properties, i.e. the StatefulOperatorStateInfo and the output mode, are initially undefined when StreamingGlobalLimitExec is created. StreamingGlobalLimitExec is updated to hold execution-specific configuration when IncrementalExecution is requested to prepare the logical plan (of a streaming query) for execution (when the state preparation rule is executed).

Creating StreamingGlobalLimitExec Instance

StreamingGlobalLimitExec takes the following to be created:

StreamingGlobalLimitExec initializes the internal properties.

StreamingGlobalLimitExec as StateStoreWriter

Performance Metrics

StreamingGlobalLimitExec uses the performance metrics of the parent StateStoreWriter.

Executing Physical Operator (Generating RDD[InternalRow]) — doExecute Method

doExecute(): RDD[InternalRow]
Note
doExecute is part of SparkPlan Contract to generate the runtime representation of an physical operator as a recipe for distributed computation over internal binary rows on Apache Spark (RDD[InternalRow]).

doExecute…​FIXME

Internal Properties

Name Description

keySchema

FIXME

Used when…​FIXME

valueSchema

FIXME

Used when…​FIXME

results matching ""

    No results matching ""