SQLMetric — SQL Execution Metric of Physical Operator

SQLMetric is a SQL metric for monitoring execution of a physical operator.

SQLMetric is an accumulator (and that is the mechanism to propagate SQL metric updates on the executors to the driver, e.g. web UI).

Note
 Use Details for Query page in SQL tab in web UI to see the SQL execution metrics of a structured query.
Note

SQL metrics are collected using SparkListener. If there are no tasks, Spark SQL cannot collect any metrics. Updates to metrics on the driver-side require explicit call of SQLMetrics.postDriverMetricUpdates.

This is why executing some physical operators (e.g. LocalTableScanExec) may not have SQL metrics in web UI’s Details for Query in SQL tab.

Compare the following SQL queries and their execution pages.

// The query does not have SQL metrics in web UI
Seq("Jacek").toDF("name").show

// The query gives numOutputRows metric in web UI's Details for Query (SQL tab)
Seq("Jacek").toDF("name").count

SQLMetric takes a metric type and an initial value when created.

Table 1. Metric Types and Corresponding Create Methods
Metric Type Create Method Failed Values Counted? Description

size

createSizeMetric

no

Used when…​

sum

createMetric

no

Used when…​

timing

createTimingMetric

no

Used when…​

reset Method

reset(): Unit

reset…​FIXME

Note
 reset is used when…​FIXME

Posting Driver-Side Metric Updates — SQLMetrics.postDriverMetricUpdates Method

postDriverMetricUpdates(
  sc: SparkContext,
  executionId: String,
  metrics: Seq[SQLMetric]): Unit

postDriverMetricUpdates posts a SparkListenerDriverAccumUpdates event to LiveListenerBus when executionId is specified.

Note
 postDriverMetricUpdates method belongs to SQLMetrics object.
Note

postDriverMetricUpdates is used when:

results matching ""

    No results matching ""