RewriteCorrelatedScalarSubquery Logical Optimization

RewriteCorrelatedScalarSubquery is a base logical optimization that transforms logical plans with the following operators:

  1. FIXME

RewriteCorrelatedScalarSubquery is part of the Operator Optimization before Inferring Filters fixed-point batch in the standard batches of the Catalyst Optimizer.

RewriteCorrelatedScalarSubquery is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].

import org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery

// Demo: Filter + Aggregate
// Demo: Filter + UnaryNode

val plan = ???
val optimizedPlan = RewriteCorrelatedScalarSubquery(plan)

evalExpr Internal Method

evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) : Option[Any]


evalExpr is used exclusively when RewriteCorrelatedScalarSubquery is…​FIXME

evalAggOnZeroTups Internal Method

evalAggOnZeroTups(expr: Expression) : Option[Any]


evalAggOnZeroTups is used exclusively when RewriteCorrelatedScalarSubquery is…​FIXME

evalSubqueryOnZeroTups Internal Method

evalSubqueryOnZeroTups(plan: LogicalPlan) : Option[Any]


evalSubqueryOnZeroTups is used exclusively when RewriteCorrelatedScalarSubquery is requsted to constructLeftJoins.

constructLeftJoins Internal Method

  child: LogicalPlan,
  subqueries: ArrayBuffer[ScalarSubquery]): LogicalPlan


constructLeftJoins is used exclusively when RewriteCorrelatedScalarSubquery logical optimization is executed (i.e. applied to Aggregate, Project or Filter logical operators with correlated scalar subqueries)

Executing Rule — apply Method

apply(plan: LogicalPlan): LogicalPlan
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).

apply transforms the input logical plan as follows:

  1. For Aggregate operators, apply…​FIXME

  2. For Project operators, apply…​FIXME

  3. For Filter operators, apply…​FIXME

Extracting ScalarSubquery Expressions with Children — extractCorrelatedScalarSubqueries Internal Method

extractCorrelatedScalarSubqueries[E <: Expression](
  expression: E,
  subqueries: ArrayBuffer[ScalarSubquery]): E

extractCorrelatedScalarSubqueries finds all ScalarSubquery expressions with at least one child in the input expression and adds them to the input subqueries collection.

extractCorrelatedScalarSubqueries traverses the input expression down (the expression tree) and, every time a ScalarSubquery with at least one child is found, returns the head of the output attributes of the subquery plan.

In the end, extractCorrelatedScalarSubqueries returns the rewritten expression.

extractCorrelatedScalarSubqueries uses scala.collection.mutable.ArrayBuffer and mutates an instance inside (i.e. adds ScalarSubquery expressions) that makes for two output values, i.e. the rewritten expression and the ScalarSubquery expressions.
extractCorrelatedScalarSubqueries is used exclusively when RewriteCorrelatedScalarSubquery is executed (i.e. applied to a logical plan).

results matching ""

    No results matching ""