BaseSessionStateBuilder — Generic Builder of SessionState

BaseSessionStateBuilder is the abstraction of builders that can produce a new BaseSessionStateBuilder to create a SessionState.

Note
 BaseSessionStateBuilder and spark.sql.catalogImplementation configuration property allow for Hive and non-Hive Spark deployments.
Table 1. BaseSessionStateBuilder Contract (Abstract Methods Only)
Method Description

newBuilder

 
newBuilder: (SparkSession, Option[SessionState]) => BaseSessionStateBuilder

Produces a new BaseSessionStateBuilder for given SparkSession and optional SessionState

Used when BaseSessionStateBuilder is requested to create a SessionState

BaseSessionStateBuilder is created when SparkSession is requested for a SessionState.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState
org.apache.spark.sql.internal.SessionState

BaseSessionStateBuilder holds properties that (together with newBuilder) are used to create a SessionState.

Table 2. BaseSessionStateBuilder’s Properties
Name Description

analyzer

Logical analyzer

catalog

SessionCatalog

Used to create Analyzer, Optimizer and a SessionState itself

Note
 HiveSessionStateBuilder manages its own Hive-aware HiveSessionCatalog.

conf

SQLConf

customOperatorOptimizationRules

Custom operator optimization rules to add to the base Operator Optimization batch.

When requested for the custom rules, customOperatorOptimizationRules simply requests the SparkSessionExtensions to buildOptimizerRules.

experimentalMethods

ExperimentalMethods

extensions

SparkSessionExtensions

functionRegistry

FunctionRegistry

listenerManager

ExecutionListenerManager

optimizer

SparkOptimizer (that is downcast to the base Optimizer) that is created with the SessionCatalog and the ExperimentalMethods.

Note that the SparkOptimizer adds the customOperatorOptimizationRules to the operator optimization rules.

optimizer is used when BaseSessionStateBuilder is requested to create a SessionState (for the optimizerBuilder function to create an Optimizer when requested for the Optimizer).

planner

SparkPlanner

resourceLoader

SessionResourceLoader

sqlParser

ParserInterface

streamingQueryManager

Spark Structured Streaming’s StreamingQueryManager

udfRegistration

UDFRegistration
Table 3. BaseSessionStateBuilders
BaseSessionStateBuilder Description

SessionStateBuilder

HiveSessionStateBuilder

Note

BaseSessionStateBuilder defines a type alias NewBuilder for a function to create a BaseSessionStateBuilder.

type NewBuilder = (SparkSession, Option[SessionState]) => BaseSessionStateBuilder
Note
 BaseSessionStateBuilder is an experimental and unstable API.

Creating BaseSessionStateBuilder Instance

BaseSessionStateBuilder takes the following to be created:

Creating Clone of SessionState (Lazily) — createClone Method

createClone: (SparkSession, SessionState) => SessionState

createClone creates a SessionState (lazily as a function) using newBuilder followed by build.

Note
 createClone is used when BaseSessionStateBuilder is requested for a SessionState.

Creating SessionState — build Method

build(): SessionState

build creates a SessionState with the following:

Note

build is used when:

Getting Function to Create QueryExecution For LogicalPlan — createQueryExecution Method

createQueryExecution: LogicalPlan => QueryExecution

createQueryExecution simply returns a function that takes a LogicalPlan and creates a QueryExecution with the SparkSession and the logical plan.

Note
 createQueryExecution is used exclusively when BaseSessionStateBuilder is requested to create a SessionState instance.

results matching ""

    No results matching ""