Powered by GitBook

WindowExpression Unevaluable Expression

WindowExpression is an unevaluable expression that represents a window function (over some WindowSpecDefinition).

Note	An unevaluable expression cannot be evaluated to produce a value (neither in interpreted nor code-generated expression evaluations) and has to be resolved (replaced) to some other expressions or logical operators at analysis or optimization phases or they fail analysis.

WindowExpression is created when:

WindowSpec is requested to withAggregate (when Column.over operator is used)
WindowsSubstitution logical evaluation rule is executed (with WithWindowDefinition logical operators with UnresolvedWindowExpression expressions)
AstBuilder is requested to parse a function call in a SQL statement

Note	`WindowExpression` can only be created with AggregateExpression, AggregateWindowFunction or OffsetWindowFunction expressions which is enforced at analysis.

// Using Catalyst DSL
val wf = 'count.function(star())
val windowSpec = ???

Note	`WindowExpression` is resolved in ExtractWindowExpressions, ResolveWindowFrame and ResolveWindowOrder logical rules.

import org.apache.spark.sql.catalyst.expressions.WindowExpression
// relation - Dataset as a table to query
val table = spark.emptyDataset[Int]

scala> val windowExpr = table
  .selectExpr("count() OVER (PARTITION BY value) AS count")
  .queryExecution
  .logical      (1)
  .expressions
  .toList(0)
  .children(0)
  .asInstanceOf[WindowExpression]
windowExpr: org.apache.spark.sql.catalyst.expressions.WindowExpression = 'count() windowspecdefinition('value, UnspecifiedFrame)

scala> windowExpr.sql
res2: String = count() OVER (PARTITION BY `value` UnspecifiedFrame)

Use sqlParser directly as in WithWindowDefinition Example

Table 1. WindowExpression’s Properties
Name	Description
`children`	Collection of two expressions, i.e. windowFunction and WindowSpecDefinition, for which `WindowExpression` was created.
`dataType`	DataType of windowFunction
`foldable`	Whether or not windowFunction is foldable.
`nullable`	Whether or not windowFunction is nullable.
`sql`	`"[windowFunction].sql OVER [windowSpec].sql"`
`toString`	`"[windowFunction] [windowSpec]"`

Note	`WindowExpression` is subject to NullPropagation and DecimalAggregates logical optimizations.

Note	Distinct window functions are not supported which is enforced at analysis.

Note	An offset window function can only be evaluated in an ordered row-based window frame with a single offset which is enforced at analysis.

Catalyst DSL — `windowExpr` Operator

windowExpr(
  windowFunc: Expression,
  windowSpec: WindowSpecDefinition): WindowExpression

windowExpr operator in Catalyst DSL creates a WindowExpression expression, e.g. for testing or Spark SQL internals exploration.

// FIXME: DEMO

Creating WindowExpression Instance

WindowExpression takes the following when created:

Window function expression
WindowSpecDefinition expression

results matching ""

No results matching ""