Executors Tab

Executors tab in web UI shows…​FIXME

spark webui executors.png
Figure 1. Executors Tab in web UI (local mode)

The Executors tab is available under /executors URL, i.e. http://localhost:4040/executors.

Internally, the Executors tab is represented by ExecutorsTab.

Note

What’s interesting in how Storage Memory is displayed in the Executors tab is that the default UnifiedMemoryManager calculates the maximum memory in a way that is different from what the page displays (using the custom JavaScript formatBytes function in utils.js).

// local mode with spark.driver.memory 2g
// ./bin/spark-shell --conf spark.driver.memory=2g
// UnifiedMemoryManager reports 912MB
// You can see it after enabling INFO messages for BlockManagerMasterEndpoint

INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.8:54503 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8, 54503, None)

// custom JavaScript `formatBytes` function (from utils.js) reports...956.6MB
// See https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48
def formatBytes(bytes: Double) = {
  val k = 1000
  val i = math.floor(math.log(bytes) / math.log(k))
  val maxMemoryWebUI = bytes / math.pow(k, i)
  f"$maxMemoryWebUI%1.1f"
}
scala> println(formatBytes(maxMemory))
956.6

getExecInfo Method

getExecInfo(
  listener: ExecutorsListener,
  statusId: Int,
  isActive: Boolean): ExecutorSummary

getExecInfo creates a ExecutorSummary.

Caution
FIXME
Note
getExecInfo is used when…​FIXME

Settings

spark.ui.threadDumpsEnabled

spark.ui.threadDumpsEnabled (default: true) is to enable (true) or disable (false) ExecutorThreadDumpPage.

results matching ""

    No results matching ""