GenerateUnsafeProjection is a CodeGenerator that generates the bytecode for a UnsafeProjection for given expressions (i.e. CodeGenerator[Seq[Expression], UnsafeProjection]).

GenerateUnsafeProjection: Seq[Expression] => UnsafeProjection

Enable DEBUG logging level for org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection logger to see what happens inside.

Add the following line to conf/

Refer to Logging.

Generating UnsafeProjection — generate Method

  expressions: Seq[Expression],
  subexpressionEliminationEnabled: Boolean): UnsafeProjection

generate canonicalize the input expressions followed by generating a JVM bytecode for a UnsafeProjection for the expressions.


generate is used when:

canonicalize Method

canonicalize(in: Seq[Expression]): Seq[Expression]

canonicalize removes unnecessary Alias expressions.

Internally, canonicalize uses ExpressionCanonicalizer rule executor (that in turn uses just one CleanExpressions expression rule).

Generating JVM Bytecode For UnsafeProjection For Given Expressions (With Optional Subexpression Elimination) — create Method

  expressions: Seq[Expression],
  subexpressionEliminationEnabled: Boolean): UnsafeProjection
create(references: Seq[Expression]): UnsafeProjection (1)
  1. Calls the former create with subexpressionEliminationEnabled flag off

create first creates a CodegenContext and an Java source code for the input expressions.

create creates a code body with public java.lang.Object generate(Object[] references) method that creates a SpecificUnsafeProjection.

public java.lang.Object generate(Object[] references) {
  return new SpecificUnsafeProjection(references);

class SpecificUnsafeProjection extends UnsafeProjection {

create creates a CodeAndComment with the code body and comment placeholders.

You should see the following DEBUG message in the logs:

DEBUG GenerateUnsafeProjection: code for [expressions]:

Enable DEBUG logging level for org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator logger to see the message above.

create requests CodegenContext for references and requests the compiled class to create a SpecificUnsafeProjection for the input references that in the end is the final UnsafeProjection.

(Single-argument) create is part of CodeGenerator Contract.

Creating ExprCode for Expressions (With Optional Subexpression Elimination) — createCode Method

  ctx: CodegenContext,
  expressions: Seq[Expression],
  useSubexprElimination: Boolean = false): ExprCode

createCode requests the input CodegenContext to generate a Java source code for code-generated evaluation of every expression in the input expressions.


import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext

// Use Catalyst DSL
import org.apache.spark.sql.catalyst.dsl.expressions._
val expressions = "hello""world") :: "hello""world") :: Nil

import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection
val eval = GenerateUnsafeProjection.createCode(ctx, expressions, useSubexprElimination = true)

scala> println(eval.code)


        mutableStateArray2[0].write(0, ((UTF8String) references[0] /* literal */));

            mutableStateArray2[0].write(1, ((UTF8String) references[1] /* literal */));

scala> println(eval.value)

createCode is used when:

results matching ""

    No results matching ""