ExtractWindowExpressions Logical Resolution Rule

ExtractWindowExpressions is a logical resolution rule that transforms a logical query plan and replaces (extracts) WindowExpression expressions with Window logical operators.

ExtractWindowExpressions is part of the Resolution fixed-point batch in the standard batches of the Analyzer.

ExtractWindowExpressions is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].

Note
 ExtractWindowExpressions is a Scala object inside Analyzer class (so you have to create an instance of the Analyzer class to access it or simply use SessionState). 
import spark.sessionState.analyzer.ExtractWindowExpressions

// Example 1: Filter + Aggregate with WindowExpressions in aggregateExprs
val q = ???
val plan = q.queryExecution.logical
val afterExtractWindowExpressions = ExtractWindowExpressions(plan)

// Example 2: Aggregate with WindowExpressions in aggregateExprs
val q = ???
val plan = q.queryExecution.logical
val afterExtractWindowExpressions = ExtractWindowExpressions(plan)

// Example 3: Project with WindowExpressions in projectList
val q = ???
val plan = q.queryExecution.logical
val afterExtractWindowExpressions = ExtractWindowExpressions(plan)

Executing Rule — apply Method

apply(plan: LogicalPlan): LogicalPlan
Note
 apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).

apply transforms the logical operators downwards in the input logical plan as follows:

hasWindowFunction Internal Method

hasWindowFunction(projectList: Seq[NamedExpression]): Boolean (1)
hasWindowFunction(expr: NamedExpression): Boolean

  1. Executes the other hasWindowFunction on every NamedExpression in the projectList

hasWindowFunction is positive (true) when the input expr named expression is a WindowExpression expression. Otherwise, hasWindowFunction is negative (false).

Note
 hasWindowFunction is used when ExtractWindowExpressions logical resolution rule is requested to extract and execute.

extract Internal Method

extract(expressions: Seq[NamedExpression]): (Seq[NamedExpression], Seq[NamedExpression])

extract…​FIXME

Note
 extract is used exclusively when ExtractWindowExpressions logical resolution rule is executed.

Adding Project and Window Logical Operators to Logical Plan — addWindow Internal Method

addWindow(
  expressionsWithWindowFunctions: Seq[NamedExpression],
  child: LogicalPlan): LogicalPlan

addWindow adds a Project logical operator with one or more Window logical operators (for every WindowExpression in the input named expressions) to the input logical plan.

Internally, addWindow…​FIXME

Note
 addWindow is used exclusively when ExtractWindowExpressions logical resolution rule is executed.

results matching ""

    No results matching ""