PullupCorrelatedPredicates Logical Optimization

PullupCorrelatedPredicates is a base logical optimization that transforms logical plans with the following operators:

  1. Filter operators with an Aggregate child operator

  2. UnaryNode operators

PullupCorrelatedPredicates is part of the Pullup Correlated Expressions once-executed batch in the standard batches of the Catalyst Optimizer.

PullupCorrelatedPredicates is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].

import org.apache.spark.sql.catalyst.optimizer.PullupCorrelatedPredicates

// Demo: Filter + Aggregate
// Demo: Filter + UnaryNode

val plan = ???
val optimizedPlan = PullupCorrelatedPredicates(plan)

PullupCorrelatedPredicates uses PredicateHelper for…​FIXME

pullOutCorrelatedPredicates Internal Method

  sub: LogicalPlan,
  outer: Seq[LogicalPlan]): (LogicalPlan, Seq[Expression])


pullOutCorrelatedPredicates is used exclusively when PullupCorrelatedPredicates is requested to rewriteSubQueries.

rewriteSubQueries Internal Method

rewriteSubQueries(plan: LogicalPlan, outerPlans: Seq[LogicalPlan]): LogicalPlan


rewriteSubQueries is used exclusively when PullupCorrelatedPredicates is executed (i.e. applied to a logical plan).

Executing Rule — apply Method

apply(plan: LogicalPlan): LogicalPlan
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).

apply transforms the input logical plan as follows:

  1. For Filter operators with an Aggregate child operator, apply rewriteSubQueries with the Filter and the Aggregate and its child as the outer plans

  2. For UnaryNode operators, apply rewriteSubQueries with the operator and its children as the outer plans

results matching ""

    No results matching ""