SparkSessionExtensions

SparkSessionExtensions is an interface that a Spark developer can use to extend a SparkSession with custom query execution rules and a relational entity parser.

As a Spark developer, you use Builder.withExtensions method (while building a new SparkSession) to access the session-bound SparkSessionExtensions.

Table 1. SparkSessionExtensions API
Method	Description
injectCheckRule	`injectCheckRule(builder: SparkSession => LogicalPlan => Unit): Unit`
injectOptimizerRule	Registering a custom operator optimization rule `injectOptimizerRule(builder: SparkSession => Rule[LogicalPlan]): Unit`
injectParser	`injectParser(builder: (SparkSession, ParserInterface) => ParserInterface): Unit`
injectPlannerStrategy	`injectPlannerStrategy(builder: SparkSession => Strategy): Unit`
injectPostHocResolutionRule	`injectPostHocResolutionRule(builder: SparkSession => Rule[LogicalPlan]): Unit`
injectResolutionRule	`injectResolutionRule(builder: SparkSession => Rule[LogicalPlan]): Unit`

SparkSessionExtensions is an integral part of SparkSession (and is indirectly required to create one).

Table 2. SparkSessionExtensions’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`optimizerRules`	Collection of `RuleBuilder` functions (i.e. `SparkSession ⇒ Rule[LogicalPlan]`) Used when `SparkSessionExtensions` is requested to: Associate custom operator optimization rules with SparkSession Register a custom operator optimization rule

Associating Custom Operator Optimization Rules with SparkSession — `buildOptimizerRules` Method

buildOptimizerRules(session: SparkSession): Seq[Rule[LogicalPlan]]

buildOptimizerRules gives the optimizerRules logical rules that are associated with the input SparkSession.

Note	`buildOptimizerRules` is used exclusively when `BaseSessionStateBuilder` is requested for the custom operator optimization rules to add to the base Operator Optimization batch.

Registering Custom Check Analysis Rule (Builder) — `injectCheckRule` Method

injectCheckRule(builder: SparkSession => LogicalPlan => Unit): Unit

injectCheckRule…FIXME

Registering Custom Operator Optimization Rule (Builder) — `injectOptimizerRule` Method

injectOptimizerRule(builder: SparkSession => Rule[LogicalPlan]): Unit

injectOptimizerRule simply registers a custom operator optimization rule (as a RuleBuilder function) to the optimizerRules internal registry.

Registering Custom Parser (Builder) — `injectParser` Method

injectParser(builder: (SparkSession, ParserInterface) => ParserInterface): Unit

injectParser…FIXME

Registering Custom Planner Strategy (Builder) — `injectPlannerStrategy` Method

injectPlannerStrategy(builder: SparkSession => Strategy): Unit

injectPlannerStrategy…FIXME

Registering Custom Post-Hoc Resolution Rule (Builder) — `injectPostHocResolutionRule` Method

injectPostHocResolutionRule(builder: SparkSession => Rule[LogicalPlan]): Unit

injectPostHocResolutionRule…FIXME

Registering Custom Resolution Rule (Builder) — `injectResolutionRule` Method

injectResolutionRule(builder: SparkSession => Rule[LogicalPlan]): Unit

injectResolutionRule…FIXME

SparkSessionExtensions