SQLConf — Internal Configuration Store

SQLConf is an internal key-value configuration store for parameters and hints used to configure a Spark Structured Streaming application (and Spark SQL applications in general).

The parameters and hints are accessible as property accessor methods.

SQLConf is available as the conf property of the SessionState of a SparkSession.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.conf
org.apache.spark.sql.internal.SQLConf

Table 1. SQLConf’s Property Accessor Methods
Method Name / Property	Description
`continuousStreamingExecutorQueueSize` spark.sql.streaming.continuous.executorQueueSize	Used when: `DataSourceV2ScanExec` leaf physical operator is requested for the input RDDs (and creates a ContinuousDataSourceRDD) `ContinuousCoalesceExec` unary physical operator is requested to execute
`continuousStreamingExecutorPollIntervalMs` spark.sql.streaming.continuous.executorPollIntervalMs	Used exclusively when `DataSourceV2ScanExec` leaf physical operator is requested for the input RDDs (and creates a ContinuousDataSourceRDD)
`disabledV2StreamingMicroBatchReaders` spark.sql.streaming.disabledV2MicroBatchReaders	Used exclusively when `MicroBatchExecution` is requested for the analyzed logical plan (of a streaming query)
`fileSourceLogDeletion` spark.sql.streaming.fileSource.log.deletion	Used exclusively when `FileStreamSourceLog` is requested for the isDeletingExpiredLog
`fileSourceLogCleanupDelay` spark.sql.streaming.fileSource.log.cleanupDelay	Used exclusively when `FileStreamSourceLog` is requested for the fileCleanupDelayMs
`fileSourceLogCompactInterval` spark.sql.streaming.fileSource.log.compactInterval	Used exclusively when `FileStreamSourceLog` is requested for the default compaction interval
`FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION` spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion	Used when: FlatMapGroupsWithStateStrategy execution planning strategy is requested to plan a streaming query (and creates a FlatMapGroupsWithStateExec physical operator for every FlatMapGroupsWithState logical operator) Among the checkpointed properties
`minBatchesToRetain` spark.sql.streaming.minBatchesToRetain	Used when: `CompactibleFileStreamLog` is created `StreamExecution` is created `StateStoreConf` is created
`SHUFFLE_PARTITIONS` `spark.sql.shuffle.partitions`	See spark.sql.shuffle.partitions in The Internals of Spark SQL.
`stateStoreMinDeltasForSnapshot` spark.sql.streaming.stateStore.minDeltasForSnapshot	Used (as StateStoreConf.minDeltasForSnapshot) exclusively when `HDFSBackedStateStoreProvider` is requested to doSnapshot
`stateStoreProviderClass` spark.sql.streaming.stateStore.providerClass	Used when: `StateStoreWriter` is requested to stateStoreCustomMetrics (when `StateStoreWriter` is requested for the metrics and getProgress) `StateStoreConf` is created
`STREAMING_AGGREGATION_STATE_FORMAT_VERSION` spark.sql.streaming.aggregation.stateFormatVersion	Used when: StatefulAggregationStrategy execution planning strategy is executed `OffsetSeqMetadata` is requested for the relevantSQLConfs and the relevantSQLConfDefaultValues
`STREAMING_CHECKPOINT_FILE_MANAGER_CLASS` spark.sql.streaming.checkpointFileManagerClass	Used exclusively when `CheckpointFileManager` helper object is requested to create a CheckpointFileManager
`streamingMetricsEnabled` spark.sql.streaming.metricsEnabled	Used exclusively when `StreamExecution` is requested for runStream (to control whether to register a metrics reporter for a streaming query)
`STREAMING_MULTIPLE_WATERMARK_POLICY` spark.sql.streaming.multipleWatermarkPolicy
`streamingNoDataMicroBatchesEnabled` spark.sql.streaming.noDataMicroBatches.enabled	Used exclusively when `MicroBatchExecution` stream execution engine is requested to run an activated streaming query
`streamingNoDataProgressEventInterval` spark.sql.streaming.noDataProgressEventInterval	Used exclusively for ProgressReporter
`streamingPollingDelay` spark.sql.streaming.pollingDelay	Used exclusively when `StreamExecution` is created
`streamingProgressRetention` spark.sql.streaming.numRecentProgressUpdates	Used exclusively when `ProgressReporter` is requested to update progress of streaming query (and possibly remove an excess)

SQLConf

SQLConf — Internal Configuration Store

results matching ""

No results matching ""