SortExec Unary Physical Operator

SortExec is a unary physical operator that is created when:

SortExec supports Java code generation (aka codegen).

val q = Seq((0, "zero"), (1, "one")).toDF("id", "name").sort('id)
val qe = q.queryExecution

val logicalPlan = qe.analyzed
scala> println(logicalPlan.numberedTreeString)
00 Sort [id#72 ASC NULLS FIRST], true
01 +- Project [_1#69 AS id#72, _2#70 AS name#73]
02    +- LocalRelation [_1#69, _2#70]

// BasicOperators does the conversion of Sort logical operator to SortExec
val sparkPlan = qe.sparkPlan
scala> println(sparkPlan.numberedTreeString)
00 Sort [id#72 ASC NULLS FIRST], true, 0
01 +- LocalTableScan [id#72, name#73]

// SortExec supports Whole-Stage Code Generation
val executedPlan = qe.executedPlan
scala> println(executedPlan.numberedTreeString)
00 *(1) Sort [id#72 ASC NULLS FIRST], true, 0
01 +- Exchange rangepartitioning(id#72 ASC NULLS FIRST, 200)
02    +- LocalTableScan [id#72, name#73]

import org.apache.spark.sql.execution.SortExec
val sortExec = executedPlan.collect { case se: SortExec => se }.head

When requested for the output attributes, SortExec simply gives whatever the child operator uses.

When requested for the output data partitioning requirements, SortExec simply gives whatever the child operator uses.

When requested for the required partition requirements, SortExec gives the OrderedDistribution (with the sorting order expressions for the ordering) when the global flag is enabled (true) or the UnspecifiedDistribution.

SortExec operator uses the spark.sql.sort.enableRadixSort internal configuration property (enabled by default) to control…​FIXME

Table 1. SortExec’s Performance Metrics
Key Name (in web UI) Description


peak memory


sort time


spill size

Generating Java Source Code for Produce Path in Whole-Stage Code Generation — doProduce Method

doProduce(ctx: CodegenContext): String
doProduce is part of CodegenSupport Contract to generate the Java source code for produce path in Whole-Stage Code Generation.


Creating SortExec Instance

SortExec takes the following when created:

createSorter Method

createSorter(): UnsafeExternalRowSorter


createSorter is used when…​FIXME

results matching ""

    No results matching ""