BaseSessionStateBuilder — Generic Builder of SessionState

BaseSessionStateBuilder is the contract of builder objects that coordinate construction of a new SessionState.

Table 1. BaseSessionStateBuilders
BaseSessionStateBuilder Description

SessionStateBuilder

HiveSessionStateBuilder

BaseSessionStateBuilder is created when SparkSession is requested for a SessionState.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState
org.apache.spark.sql.internal.SessionState

BaseSessionStateBuilder requires that implementations define newBuilder method that SparkSession uses (indirectly) when requested for the SessionState (per spark.sql.catalogImplementation internal configuration property).

newBuilder: (SparkSession, Option[SessionState]) => BaseSessionStateBuilder
Note
BaseSessionStateBuilder and spark.sql.catalogImplementation configuration property allow for Hive and non-Hive Spark deployments.

BaseSessionStateBuilder holds properties that (together with newBuilder) are used to create a SessionState.

Table 2. BaseSessionStateBuilder’s Properties
Name Description

analyzer

Logical analyzer

catalog

Used to create Analyzer, Optimizer and a SessionState itself

Note
HiveSessionStateBuilder manages its own Hive-aware HiveSessionCatalog.

conf

SQLConf

customOperatorOptimizationRules

Custom operator optimization rules to add to the base Operator Optimization batch.

When requested for the custom rules, customOperatorOptimizationRules simply requests the SparkSessionExtensions to buildOptimizerRules.

experimentalMethods

ExperimentalMethods

extensions

SparkSessionExtensions

functionRegistry

FunctionRegistry

listenerManager

ExecutionListenerManager

optimizer

SparkOptimizer (that is downcast to the base Optimizer) that is created with the SessionCatalog and the ExperimentalMethods.

Note that the SparkOptimizer adds the customOperatorOptimizationRules to the operator optimization rules.

optimizer is used when BaseSessionStateBuilder is requested to create a SessionState (for the optimizerBuilder function to create an Optimizer when requested for the Optimizer).

planner

SparkPlanner

resourceLoader

SessionResourceLoader

sqlParser

ParserInterface

streamingQueryManager

Spark Structured Streaming’s StreamingQueryManager

udfRegistration

UDFRegistration

Note

BaseSessionStateBuilder defines a type alias NewBuilder for a function to create a BaseSessionStateBuilder.

type NewBuilder = (SparkSession, Option[SessionState]) => BaseSessionStateBuilder
Note
BaseSessionStateBuilder is an experimental and unstable API.

Creating Function to Build SessionState — createClone Method

createClone: (SparkSession, SessionState) => SessionState

createClone gives a function of SparkSession and SessionState that executes newBuilder followed by build.

Note
createClone is used exclusively when BaseSessionStateBuilder is requested for a SessionState

Creating SessionState Instance — build Method

build(): SessionState

build creates a SessionState with the following:

Note

build is used when:

Creating BaseSessionStateBuilder Instance

BaseSessionStateBuilder takes the following when created:

Getting Function to Create QueryExecution For LogicalPlan — createQueryExecution Method

createQueryExecution: LogicalPlan => QueryExecution

createQueryExecution simply returns a function that takes a LogicalPlan and creates a QueryExecution with the SparkSession and the logical plan.

Note
createQueryExecution is used exclusively when BaseSessionStateBuilder is requested to create a SessionState instance.

results matching ""

    No results matching ""