SparkPlanner — Spark Query Planner

SparkPlanner is a concrete Catalyst Query Planner that converts a logical plan to one or more physical plans using execution planning strategies with support for extra strategies (by means of ExperimentalMethods) and extraPlanningStrategies.

SparkPlanner is expected to plan (aka generate) at least one physical plan per logical plan.

SparkPlanner is available as planner of a SessionState.

val spark: SparkSession = ...
scala> :type spark.sessionState.planner
Table 1. SparkPlanner’s Execution Planning Strategies (in execution order)
SparkStrategy Description

ExperimentalMethods's extraStrategies


Extension point for extra planning strategies









SparkPlanner extends SparkStrategies abstract class.

Creating SparkPlanner Instance

SparkPlanner takes the following when created:


SparkPlanner is created in:

Extension Point for Extra Planning Strategies — extraPlanningStrategies Method

extraPlanningStrategies: Seq[Strategy] = Nil

extraPlanningStrategies is an extension point to register extra planning strategies with the query planner.

extraPlanningStrategies are executed after extraStrategies.

extraPlanningStrategies is used when SparkPlanner is requested for planning strategies.

extraPlanningStrategies is overriden in the SessionState builders — BaseSessionStateBuilder and HiveSessionStateBuilder.

Collecting PlanLater Physical Operators — collectPlaceholders Method

collectPlaceholders(plan: SparkPlan): Seq[(SparkPlan, LogicalPlan)]

collectPlaceholders collects all PlanLater physical operators in the plan physical plan.

collectPlaceholders is part of QueryPlanner Contract.

Pruning "Bad" Physical Plans — prunePlans Method

prunePlans(plans: Iterator[SparkPlan]): Iterator[SparkPlan]

prunePlans gives the input plans physical plans back (i.e. with no changes).

prunePlans is part of QueryPlanner Contract to remove somehow "bad" plans.

Creating Physical Operator (Possibly Under FilterExec and ProjectExec Operators) — pruneFilterProject Method

  projectList: Seq[NamedExpression],
  filterPredicates: Seq[Expression],
  prunePushedDownFilters: Seq[Expression] => Seq[Expression],
  scanBuilder: Seq[Attribute] => SparkPlan): SparkPlan
pruneFilterProject is almost like DataSourceStrategy.pruneFilterProjectRaw.

pruneFilterProject branches off per whether it is possible to use a column pruning only (to get the right projection) and the input projectList columns of this projection are enough to evaluate all input filterPredicates filter conditions.

If so, pruneFilterProject does the following:

  1. Applies the input scanBuilder function to the input projectList columns that creates a new physical operator

  2. If there are Catalyst predicate expressions in the input prunePushedDownFilters that cannot be pushed down, pruneFilterProject creates a FilterExec unary physical operator (with the unhandled predicate expressions)

  3. Otherwise, pruneFilterProject simply returns the physical operator

In this case no extra ProjectExec unary physical operator is created.

If not (i.e. it is neither possible to use a column pruning only nor evaluate filter conditions), pruneFilterProject does the following:

  1. Applies the input scanBuilder function to the projection and filtering columns that creates a new physical operator

  2. Creates a FilterExec unary physical operator (with the unhandled predicate expressions if available)

  3. Creates a ProjectExec unary physical operator with the optional FilterExec operator (with the scan physical operator) or simply the scan physical operator alone


pruneFilterProject is used when:

results matching ""

    No results matching ""