// HACK: Disable symbolToColumn implicit conversion
// It is imported automatically in spark-shell (and makes demos impossible)
// implicit def symbolToColumn(s: Symbol): org.apache.spark.sql.ColumnName
trait ThatWasABadIdea
implicit def symbolToColumn(ack: ThatWasABadIdea) = ack
import org.apache.spark.sql.catalyst.dsl.expressions._
val boundRef = 'hello.string.at(4)
import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
val expressions: Seq[Expression] = Seq(Literal(1), boundRef)
import org.apache.spark.sql.catalyst.expressions.InterpretedProjection
val ip = new InterpretedProjection(expressions)
scala> println(ip)
Row => [1,input[4, string, true]]
InterpretedProjection
InterpretedProjection is a Projection that…FIXME
InterpretedProjection takes expressions when created.
InterpretedProjection is created when:
-
UserDefinedGeneratoris requested toinitializeConverters -
ConvertToLocalRelationlogical optimization is executed (to transformProjectlogical operators) -
HiveGenericUDTFis evaluated -
ScriptTransformationExecphysical operator is executed
Initializing Nondeterministic Expressions — initialize Method
initialize(partitionIndex: Int): Unit
|
Note
|
initialize is part of Projection Contract to…FIXME.
|
initialize requests Nondeterministic expressions (in expressions) to initialize with the partitionIndex.