SparkUI represents the web UI for a Spark application and Spark History Server. It is created and bound when SparkContext is created (with spark.ui.enabled enabled).

The only difference between SparkUI for a Spark application and Spark History Server is that…​FIXME

When started, SparkUI binds to appUIAddress address that you can control using SPARK_PUBLIC_DNS environment variable or Spark property.

Table 1. SparkUI’s Internal Registries and Counters
Name Description



Enable INFO logging level for org.apache.spark.ui.SparkUI logger to see what happens inside.

Add the following line to conf/

Refer to Logging.

attachTab Method


Creating SparkUI Instance

class SparkUI (
  val sc: Option[SparkContext],
  val conf: SparkConf,
  securityManager: SecurityManager,
  val environmentListener: EnvironmentListener,
  val storageStatusListener: StorageStatusListener,
  val executorsListener: ExecutorsListener,
  val jobProgressListener: JobProgressListener,
  val storageListener: StorageListener,
  val operationGraphListener: RDDOperationGraphListener,
  var appName: String,
  val basePath: String,
  val startTime: Long)
extends WebUI(securityManager,
  securityManager.getSSLOptions("ui"), SparkUI.getUIPort(conf),
  conf, basePath, "SparkUI")

When executed, SparkUI creates a StagesTab and initializes the tabs and handlers in web UI.

SparkUI is created when SparkContext is created (with spark.ui.enabled enabled). SparkUI gets the references to the owning SparkContext and the other properties, i.e. SparkConf, LiveListenerBus Event Bus, JobProgressListener, SecurityManager, appName, and startTime.

Assigning Unique Identifier of Spark Application — setAppId Method

setAppId(id: String): Unit

setAppId sets the internal appId.

setAppId is used exclusively when SparkContext is initialized.

Attaching Tabs and Context Handlers — initialize Method

initialize(): Unit

initialize also attaches ServletContextHandler handlers:

  1. /static to serve static files from org/apache/spark/ui/static directory (on CLASSPATH).

  2. Redirecting / to /jobs/ (so Jobs tab is the first tab when you open web UI).

  3. Serving /api context path (with org.apache.spark.status.api.v1 provider package) using ApiRootResource.

  4. Redirecting /stages/stage/kill to /stages/

initialize is part of the WebUI Contract and is executed when SparkUI is created.

Stopping SparkUI — stop Method

stop(): Unit

stop stops the HTTP server and prints the following INFO message to the logs:

INFO SparkUI: Stopped Spark web UI at [appUIAddress]
appUIAddress in the above INFO message is the result of appUIAddress method.

appUIAddress Method

appUIAddress: String

appUIAddress returns the entire URL of a Spark application’s web UI, including http:// scheme.

Internally, appUIAddress uses appUIHostPort.

getSparkUser Method

getSparkUser: String

getSparkUser returns the name of the user a Spark application runs as.

Internally, getSparkUser requests System property from EnvironmentListener Spark listener.

getSparkUser is only used to display the user name in Spark Jobs page

createLiveUI Method

  sc: SparkContext,
  conf: SparkConf,
  listenerBus: SparkListenerBus,
  jobProgressListener: JobProgressListener,
  securityManager: SecurityManager,
  appName: String,
  startTime: Long): SparkUI

createLiveUI creates a SparkUI for a live running Spark application.

Internally, createLiveUI simply forwards the call to create.

createLiveUI is called when SparkContext is created (and spark.ui.enabled is enabled).

createHistoryUI Method


create Factory Method

  sc: Option[SparkContext],
  conf: SparkConf,
  listenerBus: SparkListenerBus,
  securityManager: SecurityManager,
  appName: String,
  basePath: String = "",
  jobProgressListener: Option[JobProgressListener] = None,
  startTime: Long): SparkUI

create creates a SparkUI and is responsible for registering SparkListeners for SparkUI.

create creates a web UI for a running Spark application and Spark History Server.

Internally, create registers the following SparkListeners with the input listenerBus.

create then creates a SparkUI.

appUIHostPort Method

appUIHostPort: String

appUIHostPort returns the Spark application’s web UI which is the public hostname and port, excluding the scheme.

appUIAddress uses appUIHostPort and adds http:// scheme.

getAppName Method

getAppName: String

getAppName returns the name of the Spark application (of a SparkUI instance).

getAppName is used when SparkUITab is requested the application’s name.

SparkUITab — Custom WebUITab

SparkUITab is a private[spark] custom WebUITab that defines one method only, i.e. appName.

appName: String

appName returns the application’s name.

results matching ""

    No results matching ""