SQLConf — Internal Configuration Store

SQLConf is an internal key-value configuration store for parameters and hints used to configure a Spark Structured Streaming application (and Spark SQL applications in general).

The parameters and hints are accessible as property accessor methods.

SQLConf is available as the conf property of the SessionState of a SparkSession.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.conf
org.apache.spark.sql.internal.SQLConf
Table 1. SQLConf’s Property Accessor Methods
Method Name / Property Description

disabledV2StreamingMicroBatchReaders

spark.sql.streaming.disabledV2MicroBatchReaders

Used exclusively when MicroBatchExecution is requested for the analyzed logical plan (of a streaming query)

FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION

spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion

SHUFFLE_PARTITIONS

spark.sql.shuffle.partitions

See spark.sql.shuffle.partitions in The Internals of Spark SQL.

stateStoreProviderClass

spark.sql.streaming.stateStore.providerClass

Used when:

STREAMING_AGGREGATION_STATE_FORMAT_VERSION

spark.sql.streaming.aggregation.stateFormatVersion

Used when:

  • StatefulAggregationStrategy execution planning strategy is executed

  • OffsetSeqMetadata is requested for relevantSQLConfs and relevantSQLConfDefaultValues

streamingMetricsEnabled

spark.sql.streaming.metricsEnabled

Used exclusively when StreamExecution is requested for runStream (to control whether to register a metrics reporter for a streaming query)

STREAMING_MULTIPLE_WATERMARK_POLICY

spark.sql.streaming.multipleWatermarkPolicy

results matching ""

    No results matching ""