val spark: SparkSession = ...
// query uses InMemoryRelation logical operator
val q = spark.range(5).cache
val plan = q.queryExecution.optimizedPlan
scala> println(plan.numberedTreeString)
00 InMemoryRelation [id#208L], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
01 +- *Range (0, 5, step=1, splits=8)
// InMemoryScans is an internal class of SparkStrategies
import spark.sessionState.planner.InMemoryScans
val physicalPlan = InMemoryScans.apply(plan).head
scala> println(physicalPlan.numberedTreeString)
00 InMemoryTableScan [id#208L]
01 +- InMemoryRelation [id#208L], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
02 +- *Range (0, 5, step=1, splits=8)InMemoryScans Execution Planning Strategy
InMemoryScans is an execution planning strategy that plans InMemoryRelation logical operators to InMemoryTableScanExec physical operators.
InMemoryScans is part of the standard execution planning strategies of SparkPlanner.
Applying InMemoryScans Strategy to Logical Plan (Executing InMemoryScans) — apply Method
apply(plan: LogicalPlan): Seq[SparkPlan]|
Note
|
apply is part of GenericStrategy Contract to generate a collection of SparkPlans for a given logical plan.
|
apply destructures the input logical plan to a InMemoryRelation logical operator.
In the end, apply pruneFilterProject with a new InMemoryTableScanExec physical operator.