SparkUI — Web UI of Spark Application

SparkUI is the web UI of a Spark application (aka Application UI).

SparkUI is created along with the following:

spark webui SparkUI.png
Figure 1. Creating SparkUI for Live Spark Application

When created (while SparkContext is created for a live Spark application), SparkUI gets the following:

When started, SparkUI binds to appUIAddress address that you can control using SPARK_PUBLIC_DNS environment variable or spark.driver.host Spark property.

Note
With spark.ui.killEnabled configuration property turned on, SparkUI allows to kill jobs and stages (subject to SecurityManager.checkModifyPermissions permissions).

SparkUI gets an AppStatusStore that is then used for the following:

Table 1. SparkUI’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description

appId

Tip

Enable INFO logging level for org.apache.spark.ui.SparkUI logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.ui.SparkUI=INFO

Refer to Logging.

Assigning Unique Identifier of Spark Application — setAppId Method

setAppId(id: String): Unit

setAppId sets the internal appId.

Note
setAppId is used exclusively when SparkContext is initialized.

Stopping SparkUI — stop Method

stop(): Unit

stop stops the HTTP server and prints the following INFO message to the logs:

INFO SparkUI: Stopped Spark web UI at [appUIAddress]
Note
appUIAddress in the above INFO message is the result of appUIAddress method.

appUIAddress Method

appUIAddress: String

appUIAddress returns the entire URL of a Spark application’s web UI, including http:// scheme.

Internally, appUIAddress uses appUIHostPort.

Accessing Spark User — getSparkUser Method

getSparkUser: String

getSparkUser returns the name of the user a Spark application runs as.

Internally, getSparkUser requests user.name System property from EnvironmentListener Spark listener.

Note
getSparkUser is used…​FIXME

createLiveUI Method

createLiveUI(
  sc: SparkContext,
  conf: SparkConf,
  listenerBus: SparkListenerBus,
  jobProgressListener: JobProgressListener,
  securityManager: SecurityManager,
  appName: String,
  startTime: Long): SparkUI

createLiveUI creates a SparkUI for a live running Spark application.

Internally, createLiveUI simply forwards the call to create.

Note
createLiveUI is called when SparkContext is created (and spark.ui.enabled is enabled).

createHistoryUI Method

Caution
FIXME

appUIHostPort Method

appUIHostPort: String

appUIHostPort returns the Spark application’s web UI which is the public hostname and port, excluding the scheme.

Note
appUIAddress uses appUIHostPort and adds http:// scheme.

getAppName Method

getAppName: String

getAppName returns the name of the Spark application (of a SparkUI instance).

Note
getAppName is used when…​FIXME

Creating SparkUI Instance — create Factory Method

create(
  sc: Option[SparkContext],
  store: AppStatusStore,
  conf: SparkConf,
  securityManager: SecurityManager,
  appName: String,
  basePath: String = "",
  startTime: Long,
  appSparkVersion: String = org.apache.spark.SPARK_VERSION): SparkUI

create creates a SparkUI backed by a AppStatusStore.

Internally, create simply creates a new SparkUI (with the predefined Spark version).

Note

create is used when:

  • SparkContext is created (for a running Spark application)

  • FsHistoryProvider is requested to getAppUI (for a Spark application that already finished)

Creating SparkUI Instance

SparkUI takes the following when created:

Attaching Tabs and Context Handlers — initialize Method

initialize(): Unit
Note
initialize is part of WebUI Contract to initialize web components.

initialize creates and attaches the following tabs (with the reference to the SparkUI and its AppStatusStore):

In the end, initialize creates and attaches the following ServletContextHandlers:

  1. Creates a static handler for serving files from a static directory, i.e. /static to serve static files from org/apache/spark/ui/static directory (on CLASSPATH)

  2. Creates a redirect handler to redirect / to /jobs/ (and so the Jobs tab is the welcome tab when you open the web UI)

  3. Creates the /api/* context handler for the Status REST API

  4. Creates a redirect handler to redirect /jobs/job/kill to /jobs/ and request the JobsTab to execute handleKillRequest before redirection

  5. Creates a redirect handler to redirect /stages/stage/kill to /stages/ and request the StagesTab to execute handleKillRequest before redirection

results matching ""

    No results matching ""