// HACK: Disable symbolToColumn implicit conversion
// It is imported automatically in spark-shell (and makes demos impossible)
// implicit def symbolToColumn(s: Symbol): org.apache.spark.sql.ColumnName
trait ThatWasABadIdea
implicit def symbolToColumn(ack: ThatWasABadIdea) = ack
import org.apache.spark.sql.catalyst.dsl.expressions._
val boundRef = 'hello.string.at(4)
import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
val expressions: Seq[Expression] = Seq(Literal(1), boundRef)
import org.apache.spark.sql.catalyst.expressions.InterpretedProjection
val ip = new InterpretedProjection(expressions)
scala> println(ip)
Row => [1,input[4, string, true]]
InterpretedProjection
InterpretedProjection
is a Projection that…FIXME
InterpretedProjection
takes expressions when created.
InterpretedProjection
is created when:
-
UserDefinedGenerator
is requested toinitializeConverters
-
ConvertToLocalRelation
logical optimization is executed (to transformProject
logical operators) -
HiveGenericUDTF
is evaluated -
ScriptTransformationExec
physical operator is executed
Initializing Nondeterministic Expressions — initialize
Method
initialize(partitionIndex: Int): Unit
Note
|
initialize is part of Projection Contract to…FIXME.
|
initialize
requests Nondeterministic
expressions (in expressions) to initialize with the partitionIndex
.