SparkSessionExtensions

SparkSessionExtensions is an interface that a Spark developer can use to extend a SparkSession with custom query execution rules and a relational entity parser.

As a Spark developer, you use Builder.withExtensions method (while building a new SparkSession) to access the session-bound SparkSessionExtensions.

Table 1. SparkSessionExtensions API
Method Description

injectCheckRule

injectCheckRule(builder: SparkSession => LogicalPlan => Unit): Unit

injectOptimizerRule

Registering a custom operator optimization rule

injectOptimizerRule(builder: SparkSession => Rule[LogicalPlan]): Unit

injectParser

injectParser(builder: (SparkSession, ParserInterface) => ParserInterface): Unit

injectPlannerStrategy

injectPlannerStrategy(builder: SparkSession => Strategy): Unit

injectPostHocResolutionRule

injectPostHocResolutionRule(builder: SparkSession => Rule[LogicalPlan]): Unit

injectResolutionRule

injectResolutionRule(builder: SparkSession => Rule[LogicalPlan]): Unit

SparkSessionExtensions is an integral part of SparkSession (and is indirectly required to create one).

Table 2. SparkSessionExtensions’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description

optimizerRules

Collection of RuleBuilder functions (i.e. SparkSession ⇒ Rule[LogicalPlan])

Used when SparkSessionExtensions is requested to:

Associating Custom Operator Optimization Rules with SparkSession — buildOptimizerRules Method

buildOptimizerRules(session: SparkSession): Seq[Rule[LogicalPlan]]

buildOptimizerRules gives the optimizerRules logical rules that are associated with the input SparkSession.

Note
buildOptimizerRules is used exclusively when BaseSessionStateBuilder is requested for the custom operator optimization rules to add to the base Operator Optimization batch.

Registering Custom Check Analysis Rule (Builder) — injectCheckRule Method

injectCheckRule(builder: SparkSession => LogicalPlan => Unit): Unit

injectCheckRule…​FIXME

Registering Custom Operator Optimization Rule (Builder) — injectOptimizerRule Method

injectOptimizerRule(builder: SparkSession => Rule[LogicalPlan]): Unit

injectOptimizerRule simply registers a custom operator optimization rule (as a RuleBuilder function) to the optimizerRules internal registry.

Registering Custom Parser (Builder) — injectParser Method

injectParser(builder: (SparkSession, ParserInterface) => ParserInterface): Unit

injectParser…​FIXME

Registering Custom Planner Strategy (Builder) — injectPlannerStrategy Method

injectPlannerStrategy(builder: SparkSession => Strategy): Unit

injectPlannerStrategy…​FIXME

Registering Custom Post-Hoc Resolution Rule (Builder) — injectPostHocResolutionRule Method

injectPostHocResolutionRule(builder: SparkSession => Rule[LogicalPlan]): Unit

injectPostHocResolutionRule…​FIXME

Registering Custom Resolution Rule (Builder) — injectResolutionRule Method

injectResolutionRule(builder: SparkSession => Rule[LogicalPlan]): Unit

injectResolutionRule…​FIXME

results matching ""

    No results matching ""