RewriteCorrelatedScalarSubquery Logical Optimization

RewriteCorrelatedScalarSubquery is a base logical optimization that transforms logical plans with the following operators:

FIXME

RewriteCorrelatedScalarSubquery is part of the Operator Optimization before Inferring Filters fixed-point batch in the standard batches of the Catalyst Optimizer.

RewriteCorrelatedScalarSubquery is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].

import org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery

// FIXME
// Demo: Filter + Aggregate
// Demo: Filter + UnaryNode

val plan = ???
val optimizedPlan = RewriteCorrelatedScalarSubquery(plan)

`evalExpr` Internal Method

evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) : Option[Any]

evalExpr…FIXME

Note	`evalExpr` is used exclusively when `RewriteCorrelatedScalarSubquery` is…FIXME

`evalAggOnZeroTups` Internal Method

evalAggOnZeroTups(expr: Expression) : Option[Any]

evalAggOnZeroTups…FIXME

Note	`evalAggOnZeroTups` is used exclusively when `RewriteCorrelatedScalarSubquery` is…FIXME

`evalSubqueryOnZeroTups` Internal Method

evalSubqueryOnZeroTups(plan: LogicalPlan) : Option[Any]

evalSubqueryOnZeroTups…FIXME

Note	`evalSubqueryOnZeroTups` is used exclusively when `RewriteCorrelatedScalarSubquery` is requsted to constructLeftJoins.

`constructLeftJoins` Internal Method

constructLeftJoins(
  child: LogicalPlan,
  subqueries: ArrayBuffer[ScalarSubquery]): LogicalPlan

constructLeftJoins…FIXME

Note	`constructLeftJoins` is used exclusively when `RewriteCorrelatedScalarSubquery` logical optimization is executed (i.e. applied to Aggregate, Project or Filter logical operators with correlated scalar subqueries)

Executing Rule — `apply` Method

apply(plan: LogicalPlan): LogicalPlan

Note	`apply` is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).

apply transforms the input logical plan as follows:

For Aggregate operators, apply…FIXME
For Project operators, apply…FIXME
For Filter operators, apply…FIXME

Extracting ScalarSubquery Expressions with Children — `extractCorrelatedScalarSubqueries` Internal Method

extractCorrelatedScalarSubqueries[E <: Expression](
  expression: E,
  subqueries: ArrayBuffer[ScalarSubquery]): E

extractCorrelatedScalarSubqueries finds all ScalarSubquery expressions with at least one child in the input expression and adds them to the input subqueries collection.

extractCorrelatedScalarSubqueries traverses the input expression down (the expression tree) and, every time a ScalarSubquery with at least one child is found, returns the head of the output attributes of the subquery plan.

In the end, extractCorrelatedScalarSubqueries returns the rewritten expression.

Note	`extractCorrelatedScalarSubqueries` uses scala.collection.mutable.ArrayBuffer and mutates an instance inside (i.e. adds `ScalarSubquery` expressions) that makes for two output values, i.e. the rewritten expression and the `ScalarSubquery` expressions.

Note	`extractCorrelatedScalarSubqueries` is used exclusively when `RewriteCorrelatedScalarSubquery` is executed (i.e. applied to a logical plan).

RewriteCorrelatedScalarSubquery