CodegenContext

CodegenContext is…​FIXME

CodegenContext takes no input parameters.

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext

CodegenContext is created when:

CodegenContext stores expressions that don’t support codegen.

Example of CodegenContext.subexpressionElimination (through CodegenContext.generateExpressions)
import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext

// Use Catalyst DSL
import org.apache.spark.sql.catalyst.dsl.expressions._
val expressions = "hello".expr.as("world") :: "hello".expr.as("world") :: Nil

// FIXME Use a real-life query to extract the expressions

// CodegenContext.subexpressionElimination (where the elimination all happens) is a private method
// It is used exclusively in CodegenContext.generateExpressions which is public
// and does the elimination when it is enabled

// Note the doSubexpressionElimination flag is on
// Triggers the subexpressionElimination private method
ctx.generateExpressions(expressions, doSubexpressionElimination = true)

// subexpressionElimination private method uses ctx.equivalentExpressions
val commonExprs = ctx.equivalentExpressions.getAllEquivalentExprs

assert(commonExprs.length > 0, "No common expressions found")
Table 1. CodegenContext’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description

classFunctions

Mutable Scala Map with function names, their Java source code and a class name

New entries are added when CodegenContext is requested to addClass and addNewFunctionToClass

Used when CodegenContext is requested to declareAddedFunctions

equivalentExpressions

EquivalentExpressions

Expressions are added and then fetched as equivalent sets when CodegenContext is requested to subexpressionElimination (for generateExpressions with subexpression elimination enabled)

currentVars

The list of generated columns as input of current operator

INPUT_ROW

The variable name of the input row of the current operator

placeHolderToComments

Placeholders and their comments

Used when…​FIXME

references

References that are used to generate classes in the following code generators:

subExprEliminationExprs

SubExprEliminationStates by Expression

Used when…​FIXME

subexprFunctions

Names of the functions that…​FIXME

Generating Java Source Code For Code-Generated Evaluation of Multiple Expressions (With Optional Subexpression Elimination) — generateExpressions Method

generateExpressions(
  expressions: Seq[Expression],
  doSubexpressionElimination: Boolean = false): Seq[ExprCode]

(only with subexpression elimination enabled) generateExpressions does subexpressionElimination of the input expressions.

In the end, generateExpressions requests every expressions to generate the Java source code for code-generated (non-interpreted) expression evaluation.

Note

generateExpressions is used when:

addReferenceObj Method

addReferenceObj(objName: String, obj: Any, className: String = null): String

addReferenceObj…​FIXME

Note
 addReferenceObj is used when…​FIXME

subexpressionEliminationForWholeStageCodegen Method

subexpressionEliminationForWholeStageCodegen(expressions: Seq[Expression]): SubExprCodes

subexpressionEliminationForWholeStageCodegen…​FIXME

Note
 subexpressionEliminationForWholeStageCodegen is used exclusively when HashAggregateExec is requested to generate a Java source code for whole-stage consume path (with grouping keys or not).

Adding Function to Generated Class — addNewFunction Method

addNewFunction(
  funcName: String,
  funcCode: String,
  inlineToOuterClass: Boolean = false): String

addNewFunction…​FIXME

Note
 addNewFunction is used when…​FIXME

subexpressionElimination Internal Method

subexpressionElimination(expressions: Seq[Expression]): Unit

subexpressionElimination requests EquivalentExpressions to addExprTree for every expression (in the input expressions).

subexpressionElimination requests EquivalentExpressions for the equivalent sets of expressions with at least two equivalent expressions (aka common expressions).

For every equivalent expression set, subexpressionElimination does the following:

  1. Takes the first expression and requests it to generate a Java source code for the expression tree

  2. addNewFunction and adds it to subexprFunctions

  3. Creates a SubExprEliminationState and adds it with every common expression in the equivalent expression set to subExprEliminationExprs

Note
 subexpressionElimination is used exclusively when CodegenContext is requested to generateExpressions (with subexpression elimination enabled).

Adding Mutable State — addMutableState Method

addMutableState(
  javaType: String,
  variableName: String,
  initFunc: String => String = _ => "",
  forceInline: Boolean = false,
  useFreshName: Boolean = true): String

addMutableState…​FIXME

val input = ctx.addMutableState("scala.collection.Iterator", "input", v => s"$v = inputs[0];")
Note
 addMutableState is used when…​FIXME

Adding Immutable State (Unless Exists Already) — addImmutableStateIfNotExists Method

addImmutableStateIfNotExists(
  javaType: String,
  variableName: String,
  initFunc: String => String = _ => ""): Unit

addImmutableStateIfNotExists…​FIXME

val ctx: CodegenContext = ???
val partitionMaskTerm = "partitionMask"
ctx.addImmutableStateIfNotExists(ctx.JAVA_LONG, partitionMaskTerm)
Note
 addImmutableStateIfNotExists is used when…​FIXME

freshName Method

freshName(name: String): String

freshName…​FIXME

Note
 freshName is used when…​FIXME

addNewFunctionToClass Internal Method

addNewFunctionToClass(
  funcName: String,
  funcCode: String,
  className: String): mutable.Map[String, mutable.Map[String, String]]

addNewFunctionToClass…​FIXME

Note
 addNewFunctionToClass is used when…​FIXME

addClass Internal Method

addClass(className: String, classInstance: String): Unit

addClass…​FIXME

Note
 addClass is used when…​FIXME

declareAddedFunctions Method

declareAddedFunctions(): String

declareAddedFunctions…​FIXME

Note
 declareAddedFunctions is used when…​FIXME

declareMutableStates Method

declareMutableStates(): String

declareMutableStates…​FIXME

Note
 declareMutableStates is used when…​FIXME

initMutableStates Method

initMutableStates(): String

initMutableStates…​FIXME

Note
 initMutableStates is used when…​FIXME

initPartition Method

initPartition(): String

initPartition…​FIXME

Note
 initPartition is used when…​FIXME

emitExtraCode Method

emitExtraCode(): String

emitExtraCode…​FIXME

Note
 emitExtraCode is used when…​FIXME

addPartitionInitializationStatement Method

addPartitionInitializationStatement(statement: String): Unit

addPartitionInitializationStatement…​FIXME

Note
 addPartitionInitializationStatement is used when…​FIXME

results matching ""

    No results matching ""