StaticSQLConf — Cross-Session, Immutable and Static SQL Configuration

StaticSQLConf holds cross-session, immutable and static SQL configuration properties.

Table 1. StaticSQLConf’s Configuration Properties
Configuration Property

spark.sql.catalogImplementation

(internal) Configures in-memory (default) or hive-related BaseSessionStateBuilder and ExternalCatalog

Builder.enableHiveSupport is used to enable Hive support for a SparkSession.

Used when:

spark.sql.debug

(internal) Only used for internal debugging when HiveExternalCatalog is requested to restoreTableMetadata.

Default: false

Not all functions are supported when enabled.

spark.sql.extensions

Name of the SQL extension configuration class that is used to configure SparkSession extensions (when Builder is requested to get or create a SparkSession). The class should implement Function1[SparkSessionExtensions, Unit], and must have a no-args constructor.

Default: (empty)

spark.sql.filesourceTableRelationCacheSize

(internal) The maximum size of the cache that maps qualified table names to table relation plans. Must not be negative.

Default: 1000

spark.sql.globalTempDatabase

(internal) Name of the Spark-owned internal database of global temporary views

Default: global_temp

Used exclusively to create a GlobalTempViewManager when SharedState is first requested for the GlobalTempViewManager.

Note
 The name of the internal database cannot conflict with the names of any database that is already available in ExternalCatalog.

spark.sql.hive.thriftServer.singleSession

When enabled (true), Hive Thrift server is running in a single session mode. All the JDBC/ODBC connections share the temporary views, function registries, SQL configuration and the current database.

Default: false

spark.sql.queryExecutionListeners

List of class names that implement QueryExecutionListener that will be automatically registered to new SparkSessions.

Default: (empty)

The classes should have either a no-arg constructor, or a constructor that expects a SparkConf argument.

spark.sql.sources.schemaStringLengthThreshold

(internal) The maximum length allowed in a single cell when storing additional schema information in Hive’s metastore

Default: 4000

spark.sql.ui.retainedExecutions

Number of executions to retain in the Spark UI.

Default: 1000

spark.sql.warehouse.dir

Directory of a Spark warehouse

Default: spark-warehouse

The properties in StaticSQLConf can only be queried and can never be changed once the first SparkSession is created.

import org.apache.spark.sql.internal.StaticSQLConf
scala> val metastoreName = spark.conf.get(StaticSQLConf.CATALOG_IMPLEMENTATION.key)
metastoreName: String = hive

scala> spark.conf.set(StaticSQLConf.CATALOG_IMPLEMENTATION.key, "hive")
org.apache.spark.sql.AnalysisException: Cannot modify the value of a static config: spark.sql.catalogImplementation;
  at org.apache.spark.sql.RuntimeConfig.requireNonStaticConf(RuntimeConfig.scala:144)
  at org.apache.spark.sql.RuntimeConfig.set(RuntimeConfig.scala:41)
  ... 50 elided

results matching ""

    No results matching ""