Generator is an interface for Catalyst’s expressions that produce zero or more records given a single input record.

Generator is the foundation for ExplodeBase expressions.

Explode Generator Unary Expression

Explode is a unary expression that produces a sequence of records for each value in the array or map.

Explode is a result of executing explode function (in SQL and functions)

scala> sql("SELECT explode(array(10,20))").explain
== Physical Plan ==
Generate explode([10,20]), false, false, [col#68]
+- Scan OneRowRelation[]

scala> sql("SELECT explode(array(10,20))").queryExecution.optimizedPlan.expressions(0)
res18: org.apache.spark.sql.catalyst.expressions.Expression = explode([10,20])

val arrayDF = Seq(Array(0,1)).toDF("array")
scala> arrayDF.withColumn("num", explode('array)).explain
== Physical Plan ==
Generate explode(array#93), true, false, [array#93, num#102]
+- LocalTableScan [array#93]



ExplodeBase Unary Expression

ExplodeBase is the base class for Explode and PosExplode.

ExplodeBase is UnaryExpression and Generator with CodegenFallback.


results matching ""

    No results matching ""