Cannot evaluate expression: [this]UnresolvedFunction Unevaluable Expression — Logical Representation of Functions in Queries
UnresolvedFunction is an Catalyst expression that represents a function (application) in a logical query plan.
UnresolvedFunction is created as a result of the following:
-
callUDF standard function
-
RelationalGroupedDataset.agg operator with aggregation functions specified by name (that converts function names to UnresolvedFunction expressions)
-
AstBuilderis requested to visitFunctionCall (in SQL queries)
UnresolvedFunction can never be resolved (and is replaced at analysis phase).
|
Note
|
UnresolvedFunction is first looked up in LookupFunctions logical rule and then resolved in ResolveFunctions logical resolution rule.
|
Given UnresolvedFunction can never be resolved it should not come as a surprise that it cannot be evaluated either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedFunction simply reports a UnsupportedOperationException.
|
Note
|
Unevaluable expressions are expressions that have to be replaced by some other expressions during analysis or optimization (or they fail analysis). |
|
Tip
|
Use Catalyst DSL’s function or distinctFunction to create a UnresolvedFunction with isDistinct flag off and on, respectively.
|
// Using Catalyst DSL to create UnresolvedFunctions
import org.apache.spark.sql.catalyst.dsl.expressions._
// Scala Symbols supported only
val f = 'f.function()
scala> :type f
org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
scala> f.isDistinct
res0: Boolean = false
val g = 'g.distinctFunction()
scala> g.isDistinct
res1: Boolean = true Creating UnresolvedFunction (With Database Undefined) — apply Factory Method
apply(name: String, children: Seq[Expression], isDistinct: Boolean): UnresolvedFunctionapply creates a FunctionIdentifier with the name and no database first and then creates a UnresolvedFunction with the FunctionIdentifier, children and isDistinct flag.
|
Note
|
|
Creating UnresolvedFunction Instance
UnresolvedFunction takes the following when created:
-
Child expressions