PruneFileSourcePartitions Logical Optimization

PruneFileSourcePartitions is the only logical optimization rule in the Prune File Source Table Partitions batch of the SparkOptimizer.

PruneFileSourcePartitions transforms a logical query plan into a Project operator with a Filter logical operator over a "pruned" LogicalRelation with the HadoopFsRelation of a Hive partitioned table (with a PrunedInMemoryFileIndex).

Executing Rule — `apply` Method

apply(
  plan: LogicalPlan): LogicalPlan

Note	`apply` is part of the Rule contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).

apply destructures the input logical plan into a tuple of projection and filter expressions together with a leaf logical operator.

apply transforms LogicalRelations with a HadoopFsRelation and a CatalogFileIndex (i.e. for Hive tables) when there are filter expressions defined and the Hive table is partitioned.

apply resolves partition column references (by requesting the logical operator to resolve partition column attributes to concrete references in the query plan) and excludes subquery expressions.

If there are no predicates (filter expressions) left for partition pruning, apply simply does nothing more and returns the input logical query untouched.

With predicates left for partition pruning, apply requests the CatalogFileIndex for the partitions by the predicate expressions (that gives a PrunedInMemoryFileIndex for a partitioned table).

apply replaces the FileIndex in the HadoopFsRelation with the PrunedInMemoryFileIndex and the total size statistic to the PrunedInMemoryFileIndex’s.

In the end, apply creates a Filter logical operator (with the "pruned" LogicalRelation as a child operator and all the filter predicate expressions combined together with And expression) and makes it a child operator of a Project operator.

PruneFileSourcePartitions