ResolveSubquery Logical Resolution Rule

ResolveSubquery is a logical resolution that resolves subquery expressions (ScalarSubquery, Exists and In) when transforming a logical plan with the following logical operators:

  1. Filter operators with an Aggregate child operator

  2. Unary operators with the children resolved

ResolveSubquery is part of Resolution fixed-point batch of rules of the Spark Analyzer.

Technically, ResolveSubquery is a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].

// Use Catalyst DSL
import org.apache.spark.sql.catalyst.expressions._
val a = 'a.int

import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
val rel = LocalRelation(a)

import org.apache.spark.sql.catalyst.expressions.Literal
val list = Seq[Literal](1)

// FIXME Use a correct query to demo ResolveSubquery
import org.apache.spark.sql.catalyst.plans.logical.Filter
import org.apache.spark.sql.catalyst.expressions.In
val plan = Filter(condition = In(value = a, list), child = rel)

scala> println(plan.numberedTreeString)
00 Filter a#9 IN (1)
01 +- LocalRelation <empty>, [a#9]

import spark.sessionState.analyzer.ResolveSubquery
val analyzedPlan = ResolveSubquery(plan)
scala> println(analyzedPlan.numberedTreeString)
00 Filter a#9 IN (1)
01 +- LocalRelation <empty>, [a#9]

Resolving Subquery Expressions (ScalarSubquery, Exists and In) — resolveSubQueries Internal Method

resolveSubQueries(plan: LogicalPlan, plans: Seq[LogicalPlan]): LogicalPlan

resolveSubQueries requests the input logical plan to transform expressions (down the operator tree) as follows:

  1. For ScalarSubquery expressions with subquery plan not resolved and resolveSubQuery to create resolved ScalarSubquery expressions

  2. For Exists expressions with subquery plan not resolved and resolveSubQuery to create resolved Exists expressions

  3. For In expressions with ListQuery not resolved and resolveSubQuery to create resolved In expressions

Note
 resolveSubQueries is used exclusively when ResolveSubquery is executed.

resolveSubQuery Internal Method

resolveSubQuery(
  e: SubqueryExpression,
  plans: Seq[LogicalPlan])(
  f: (LogicalPlan, Seq[Expression]) => SubqueryExpression): SubqueryExpression

resolveSubQuery…​FIXME

Note
 resolveSubQuery is used exclusively when ResolveSubquery is requested to resolve subquery expressions (ScalarSubquery, Exists and In).

Applying ResolveSubquery to Logical Plan (Executing ResolveSubquery) — apply Method

apply(plan: LogicalPlan): LogicalPlan
Note
 apply is part of Rule Contract to apply a rule to a TreeNode, e.g. logical plan.

apply transforms the input logical plan as follows:

  1. For Filter operators with an Aggregate operator (as the child operator) and the children resolved, apply resolves subquery expressions (ScalarSubquery, Exists and In) with the Filter operator and the plans with the Aggregate operator and its single child

  2. For unary operators with the children resolved, apply resolves subquery expressions (ScalarSubquery, Exists and In) with the unary operator and its single child

results matching ""

    No results matching ""