CreateNamedStruct Expression

CreateNamedStruct is a CreateNamedStructLike expression that…​FIXME

CreateNamedStruct uses named_struct for the user-facing name.

// Using Catalyst DSL
import org.apache.spark.sql.catalyst.dsl.expressions._
val s = namedStruct("*")
scala> println(s)
named_struct(*)

CreateNamedStruct is registered in FunctionRegistry under the name of named_struct SQL function.

import org.apache.spark.sql.catalyst.FunctionIdentifier
val fid = FunctionIdentifier(funcName = "named_struct")
val className = spark.sessionState.functionRegistry.lookupFunction(fid).get.getClassName
scala> println(className)
org.apache.spark.sql.catalyst.expressions.CreateNamedStruct

val q = sql("SELECT named_struct('id', 0)")
// analyzed so the function is resolved already (using FunctionRegistry)
val analyzedPlan = q.queryExecution.analyzed
scala> println(analyzedPlan.numberedTreeString)
00 Project [named_struct(id, 0) AS named_struct(id, 0)#7]
01 +- OneRowRelation

val e = analyzedPlan.expressions.head.children.head
import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
assert(e.isInstanceOf[CreateNamedStruct])

CreateNamedStruct is created when:

CreateNamedStruct takes a collection of Catalyst expressions when created.

// You could also use Seq("*")
import org.apache.spark.sql.functions.lit
val exprs = Seq("a", 1).map(lit).map(_.expr)

import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
val createNamedStruct = CreateNamedStruct(exprs)

import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode}
val ctx = new CodegenContext
// doGenCode is used when Expression.genCode is executed
val ExprCode(code, _, _) = createNamedStruct.genCode(ctx)

// Helper methods
def trim(code: String): String = {
  code.trim.split("\n").map(_.trim).filter(line => line.nonEmpty).mkString("\n")
}
def prettyPrint(code: String) = println(trim(code))
// END: Helper methods

scala> println(trim(code))
Object[] values_0 = new Object[1];
if (false) {
values_0[0] = null;
} else {
values_0[0] = 1;
}
final InternalRow value_0 = new org.apache.spark.sql.catalyst.expressions.GenericInternalRow(values_0);
values_0 = null;
Tip

Use namedStruct operator from Catalyst DSL’s expressions to create a CreateNamedStruct expression.

import org.apache.spark.sql.catalyst.dsl.expressions._
val s = namedStruct()
scala> :type s
org.apache.spark.sql.catalyst.expressions.Expression

import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
assert(s.isInstanceOf[CreateNamedStruct])

val s = namedStruct("*")
scala> println(s)
named_struct(*)

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — doGenCode Method

doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode
Note
doGenCode is part of Expression Contract to generate a Java source code (ExprCode) for code-generated expression evaluation.

doGenCode…​FIXME

results matching ""

    No results matching ""