CodegenFallback Contract — Catalyst Expressions with Fallback Code Generation Mode

CodegenFallback is the contract of Catalyst expressions that do not support a Java code generation and want to fall back to interpreted mode (aka fallback mode).

CodegenFallback is used when CollapseCodegenStages physical optimization is requested to execute (and enforce whole-stage codegen requirements for Catalyst expressions).

package org.apache.spark.sql.catalyst.expressions.codegen

trait CodegenFallback extends Expression {
  // No properties (vals and methods) that have no implementation
}
Table 1. (Some Examples of) CodegenFallbacks
CodegenFallback Description

CurrentDate

CurrentTimestamp

Cube

JsonToStructs

Rollup

StructsToJson

Example — CurrentTimestamp Expression with nullable flag disabled
import org.apache.spark.sql.catalyst.expressions.CurrentTimestamp
val currTimestamp = CurrentTimestamp()

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
assert(currTimestamp.isInstanceOf[CodegenFallback], "CurrentTimestamp should be a CodegenFallback")

assert(currTimestamp.nullable == false, "CurrentTimestamp should not be nullable")

import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode}
val ctx = new CodegenContext
// doGenCode is used when Expression.genCode is executed
val ExprCode(code, _, _) = currTimestamp.genCode(ctx)

scala> println(code)

Object obj_0 = ((Expression) references[0]).eval(null);
        long value_0 = (Long) obj_0;

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — doGenCode Method

doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode
Note
 doGenCode is part of Expression Contract to generate a Java source code (ExprCode) for code-generated expression evaluation.

doGenCode requests the input CodegenContext to add itself to the references.

doGenCode walks down the expression tree to find Nondeterministic expressions and for every Nondeterministic expression does the following:

  1. Requests the input CodegenContext to add it to the references

  2. Requests the input CodegenContext to addPartitionInitializationStatement that is a Java code block as follows:

    ((Nondeterministic) references[[childIndex]])
  .initialize(partitionIndex);

In the end, doGenCode generates a plain Java source code block that is one of the following code blocks per the nullable flag. doGenCode copies the input ExprCode with the code block added (as the code property).

doGenCode Code Block for nullable flag enabled
[placeHolder]
Object [objectTerm] = ((Expression) references[[idx]]).eval([input]);
boolean [isNull] = [objectTerm] == null;
[javaType] [value] = [defaultValue];
if (![isNull]) {
  [value] = ([boxedType]) [objectTerm];
}
doGenCode Code Block for nullable flag disabled
[placeHolder]
Object [objectTerm] = ((Expression) references[[idx]]).eval([input]);
[javaType] [value] = ([boxedType]) [objectTerm];

results matching ""

    No results matching ""