WindowExpression Unevaluable Expression

WindowExpression is an unevaluable expression that represents a window function (over some WindowSpecDefinition).

Note
An unevaluable expression cannot be evaluated to produce a value (neither in interpreted nor code-generated expression evaluations) and has to be resolved (replaced) to some other expressions or logical operators at analysis or optimization phases or they fail analysis.

WindowExpression is created when:

Note
WindowExpression can only be created with AggregateExpression, AggregateWindowFunction or OffsetWindowFunction expressions which is enforced at analysis.
// Using Catalyst DSL
val wf = 'count.function(star())
val windowSpec = ???
Note
WindowExpression is resolved in ExtractWindowExpressions, ResolveWindowFrame and ResolveWindowOrder logical rules.
import org.apache.spark.sql.catalyst.expressions.WindowExpression
// relation - Dataset as a table to query
val table = spark.emptyDataset[Int]

scala> val windowExpr = table
  .selectExpr("count() OVER (PARTITION BY value) AS count")
  .queryExecution
  .logical      (1)
  .expressions
  .toList(0)
  .children(0)
  .asInstanceOf[WindowExpression]
windowExpr: org.apache.spark.sql.catalyst.expressions.WindowExpression = 'count() windowspecdefinition('value, UnspecifiedFrame)

scala> windowExpr.sql
res2: String = count() OVER (PARTITION BY `value` UnspecifiedFrame)
  1. Use sqlParser directly as in WithWindowDefinition Example

Table 1. WindowExpression’s Properties
Name Description

children

Collection of two expressions, i.e. windowFunction and WindowSpecDefinition, for which WindowExpression was created.

dataType

DataType of windowFunction

foldable

Whether or not windowFunction is foldable.

nullable

Whether or not windowFunction is nullable.

sql

"[windowFunction].sql OVER [windowSpec].sql"

toString

"[windowFunction] [windowSpec]"

Note
WindowExpression is subject to NullPropagation and DecimalAggregates logical optimizations.
Note
Distinct window functions are not supported which is enforced at analysis.
Note
An offset window function can only be evaluated in an ordered row-based window frame with a single offset which is enforced at analysis.

Catalyst DSL — windowExpr Operator

windowExpr(windowFunc: Expression, windowSpec: WindowSpecDefinition): WindowExpression

windowExpr operator in Catalyst DSL creates a WindowExpression expression, e.g. for testing or Spark SQL internals exploration.

// FIXME: DEMO

Creating WindowExpression Instance

WindowExpression takes the following when created:

results matching ""

    No results matching ""