Generate Unary Logical Operator for Lateral Views

Generate is a unary logical operator that is created to represent the following (after a logical plan is analyzed):

resolved flag is…​FIXME

Note
 resolved is part of LogicalPlan Contract to…​FIXME.

producedAttributes…​FIXME

The output schema of a Generate is…​FIXME

Note
 Generate logical operator is resolved to GenerateExec unary physical operator in BasicOperators execution planning strategy.
Tip

Use generate operator from Catalyst DSL to create a Generate logical operator, e.g. for testing or Spark SQL internals exploration.

import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.types._
val lr = LocalRelation('key.int, 'values.array(StringType))

// JsonTuple generator
import org.apache.spark.sql.catalyst.expressions.JsonTuple
import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.expressions.Expression
val children: Seq[Expression] = Seq("e")
val json_tuple = JsonTuple(children)

import org.apache.spark.sql.catalyst.dsl.plans._  // <-- gives generate
val plan = lr.generate(
  generator = json_tuple,
  join = true,
  outer = true,
  alias = Some("alias"),
  outputNames = Seq.empty)
scala> println(plan.numberedTreeString)
00 'Generate json_tuple(e), true, true, alias
01 +- LocalRelation <empty>, [key#0, values#1]

Creating Generate Instance

Generate takes the following when created:

Generate initializes the internal registries and counters.

results matching ""

    No results matching ""