ExperimentalMethods

ExperimentalMethods holds extra optimizations and strategies that are used in SparkOptimizer and SparkPlanner, respectively.

Table 1. ExperimentalMethods' Attributes
Name Description

extraOptimizations

Collection of rules to optimize LogicalPlans (i.e. Rule[LogicalPlan] objects)

extraOptimizations: Seq[Rule[LogicalPlan]]

Used when SparkOptimizer is requested for the User Provided Optimizers

extraStrategies

Collection of SparkStrategies

extraStrategies: Seq[Strategy]

Used when SessionState is requested for the SparkPlanner

ExperimentalMethods is available as the experimental property of a SparkSession.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.experimental
org.apache.spark.sql.ExperimentalMethods

Example

import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan

object SampleRule extends Rule[LogicalPlan] {
  def apply(p: LogicalPlan): LogicalPlan = p
}

scala> :type spark
org.apache.spark.sql.SparkSession

spark.experimental.extraOptimizations = Seq(SampleRule)

// extraOptimizations is used in Spark Optimizer
val rule = spark.sessionState.optimizer.batches.flatMap(_.rules).filter(_ == SampleRule).head
scala> rule.ruleName
res0: String = SampleRule

results matching ""

    No results matching ""