InterpretedProjection

InterpretedProjection is a Projection that…​FIXME

InterpretedProjection takes expressions when created.

// HACK: Disable symbolToColumn implicit conversion
// It is imported automatically in spark-shell (and makes demos impossible)
// implicit def symbolToColumn(s: Symbol): org.apache.spark.sql.ColumnName
trait ThatWasABadIdea
implicit def symbolToColumn(ack: ThatWasABadIdea) = ack

import org.apache.spark.sql.catalyst.dsl.expressions._
val boundRef = 'hello.string.at(4)

import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
val expressions: Seq[Expression] = Seq(Literal(1), boundRef)

import org.apache.spark.sql.catalyst.expressions.InterpretedProjection
val ip = new InterpretedProjection(expressions)
scala> println(ip)
Row => [1,input[4, string, true]]

InterpretedProjection is created when:

  • UserDefinedGenerator is requested to initializeConverters

  • ConvertToLocalRelation logical optimization is executed (to transform Project logical operators)

  • HiveGenericUDTF is evaluated

  • ScriptTransformationExec physical operator is executed

Initializing Nondeterministic Expressions — initialize Method

initialize(partitionIndex: Int): Unit
Note
initialize is part of Projection Contract to…​FIXME.

initialize requests Nondeterministic expressions (in expressions) to initialize with the partitionIndex.

results matching ""

    No results matching ""