SQLConf

SQLConf is a key-value configuration store for parameters and hints used in Spark SQL. It offers methods to get, set, unset or clear their values.

You can access the current SQLConf using sparkSession.conf.

Table 1. Parameters and Hints (in alphabetical order)
Name Default Value Description

spark.sql.optimizer.maxIterations

100

Maximum number of iterations for Analyzer and Optimizer.

spark.sql.streaming.fileSink.log.deletion

true

Flag to control whether to delete the expired log files in file stream sink.

spark.sql.streaming.fileSink.log.cleanupDelay

FIXME

FIXME

spark.sql.streaming.schemaInference

FIXME

FIXME

spark.sql.streaming.fileSink.log.compactInterval

FIXME

FIXME

Note
SQLConf is a private[sql] serializable class in org.apache.spark.sql.internal package.

Getting Parameters and Hints

You can get the current parameters and hints using the following family of get methods.

getConfString(key: String): String
getConf[T](entry: ConfigEntry[T], defaultValue: T): T
getConf[T](entry: ConfigEntry[T]): T
getConf[T](entry: OptionalConfigEntry[T]): Option[T]
getConfString(key: String, defaultValue: String): String
getAllConfs: immutable.Map[String, String]
getAllDefinedConfs: Seq[(String, String, String)]

Setting Parameters and Hints

You can set parameters and hints using the following family of set methods.

setConf(props: Properties): Unit
setConfString(key: String, value: String): Unit
setConf[T](entry: ConfigEntry[T], value: T): Unit

Unsetting Parameters and Hints

You can unset parameters and hints using the following family of unset methods.

unsetConf(key: String): Unit
unsetConf(entry: ConfigEntry[_]): Unit

Clearing All Parameters and Hints

clear(): Unit

You can use clear to remove all the parameters and hints in SQLConf.

results matching ""

    No results matching ""