SparkConf — Programmable Configuration for Spark Applications

Tip
Refer to Spark Configuration in the official documentation for an extensive coverage of how to configure Spark and user programs.
Caution

TODO

  • Describe SparkConf object for the application configuration.

  • the default configs

  • system properties

There are three ways to configure Spark and user programs:

  • Spark Properties - use Web UI to learn the current properties.

  • …​

setIfMissing Method

Caution
FIXME

isExecutorStartupConf Method

Caution
FIXME

set Method

Caution
FIXME

Mandatory Settings - spark.master and spark.app.name

There are two mandatory settings of any Spark application that have to be defined before this Spark application could be run — spark.master and spark.app.name.

Spark Properties

Every user program starts with creating an instance of SparkConf that holds the master URL to connect to (spark.master), the name for your Spark application (that is later displayed in web UI and becomes spark.app.name) and other Spark properties required for proper runs. The instance of SparkConf can be used to create SparkContext.

Tip

Start Spark shell with --conf spark.logConf=true to log the effective Spark configuration as INFO when SparkContext is started.

$ ./bin/spark-shell --conf spark.logConf=true
...
15/10/19 17:13:49 INFO SparkContext: Running Spark version 1.6.0-SNAPSHOT
15/10/19 17:13:49 INFO SparkContext: Spark configuration:
spark.app.name=Spark shell
spark.home=/Users/jacek/dev/oss/spark
spark.jars=
spark.logConf=true
spark.master=local[*]
spark.repl.class.uri=http://10.5.10.20:64055
spark.submit.deployMode=client
...

Use sc.getConf.toDebugString to have a richer output once SparkContext has finished initializing.

You can query for the values of Spark properties in Spark shell as follows:

scala> sc.getConf.getOption("spark.local.dir")
res0: Option[String] = None

scala> sc.getConf.getOption("spark.app.name")
res1: Option[String] = Some(Spark shell)

scala> sc.getConf.get("spark.master")
res2: String = local[*]

Setting up Spark Properties

There are the following places where a Spark application looks for Spark properties (in the order of importance from the least important to the most important):

  • conf/spark-defaults.conf - the configuration file with the default Spark properties. Read spark-defaults.conf.

  • --conf or -c - the command-line option used by spark-submit (and other shell scripts that use spark-submit or spark-class under the covers, e.g. spark-shell)

  • SparkConf

Default Configuration

The default Spark configuration is created when you execute the following code:

import org.apache.spark.SparkConf
val conf = new SparkConf

It simply loads spark.* system properties.

You can use conf.toDebugString or conf.getAll to have the spark.* system properties loaded printed out.

scala> conf.getAll
res0: Array[(String, String)] = Array((spark.app.name,Spark shell), (spark.jars,""), (spark.master,local[*]), (spark.submit.deployMode,client))

scala> conf.toDebugString
res1: String =
spark.app.name=Spark shell
spark.jars=
spark.master=local[*]
spark.submit.deployMode=client

scala> println(conf.toDebugString)
spark.app.name=Spark shell
spark.jars=
spark.master=local[*]
spark.submit.deployMode=client

Settings

Table 1. Spark Properties
Spark Property Default Value Description

spark.master

Master URL

spark.app.id

FIXME

Application Id

Set when SparkContext is created.

spark.app.name

Application Name

results matching ""

    No results matching ""