SQLConf — Internal Configuration Store

SQLConf is an internal key-value configuration store for parameters and hints used to configure a Spark Structured Streaming application (and Spark SQL applications in general).

The parameters and hints are accessible as property accessor methods.

SQLConf is available as the conf property of the SessionState of a SparkSession.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.conf
org.apache.spark.sql.internal.SQLConf
Table 1. SQLConf’s Property Accessor Methods
Method Name / Property Description

continuousStreamingExecutorQueueSize

spark.sql.streaming.continuous.executorQueueSize

Used when:

  • DataSourceV2ScanExec leaf physical operator is requested for the input RDDs (and creates a ContinuousDataSourceRDD)

  • ContinuousCoalesceExec unary physical operator is requested to execute

continuousStreamingExecutorPollIntervalMs

spark.sql.streaming.continuous.executorPollIntervalMs

Used exclusively when DataSourceV2ScanExec leaf physical operator is requested for the input RDDs (and creates a ContinuousDataSourceRDD)

disabledV2StreamingMicroBatchReaders

spark.sql.streaming.disabledV2MicroBatchReaders

Used exclusively when MicroBatchExecution is requested for the analyzed logical plan (of a streaming query)

FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION

spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion

Used when:

minBatchesToRetain

spark.sql.streaming.minBatchesToRetain

Used when:

SHUFFLE_PARTITIONS

spark.sql.shuffle.partitions

See spark.sql.shuffle.partitions in The Internals of Spark SQL.

stateStoreProviderClass

spark.sql.streaming.stateStore.providerClass

Used when:

STREAMING_AGGREGATION_STATE_FORMAT_VERSION

spark.sql.streaming.aggregation.stateFormatVersion

Used when:

streamingMetricsEnabled

spark.sql.streaming.metricsEnabled

Used exclusively when StreamExecution is requested for runStream (to control whether to register a metrics reporter for a streaming query)

STREAMING_MULTIPLE_WATERMARK_POLICY

spark.sql.streaming.multipleWatermarkPolicy

streamingNoDataProgressEventInterval

spark.sql.streaming.noDataProgressEventInterval

Used exclusively for ProgressReporter

results matching ""

    No results matching ""