CheckAnalysis — Analysis Validation

CheckAnalysis defines checkAnalysis method that Analyzer uses to check if a logical plan is correct (after all the transformations) by applying validation rules and in the end marking it as analyzed.

Note
 An analyzed logical plan is correct and ready for execution.

CheckAnalysis defines extendedCheckRules extension point that allows for extra analysis check rules.

Validating Analysis of Logical Plan (and Marking Plan As Analyzed) — checkAnalysis Method

checkAnalysis(plan: LogicalPlan): Unit

checkAnalysis recursively checks the correctness of the analysis of the input logical plan and marks it as analyzed.

Note
 checkAnalysis fails analysis when finds UnresolvedRelation in the input LogicalPlan…​FIXME What else?

Internally, checkAnalysis processes nodes in the input plan (starting from the leafs, i.e. nodes down the operator tree).

checkAnalysis skips logical plans that have already undergo analysis.

Table 1. checkAnalysis’s Validation Rules (in the order of execution)
LogicalPlan/Operator Behaviour

UnresolvedRelation

Fails analysis with the error message:

Table or view not found: [tableIdentifier]

Unresolved Attribute

Fails analysis with the error message:

cannot resolve '[expr]' given input columns: [from]

Expression with incorrect input data types

Fails analysis with the error message:

cannot resolve '[expr]' due to data type mismatch: [message]

Unresolved Cast

Fails analysis with the error message:

invalid cast from [dataType] to [dataType]

Grouping

Fails analysis with the error message:

grouping() can only be used with GroupingSets/Cube/Rollup

GroupingID

Fails analysis with the error message:

grouping_id() can only be used with GroupingSets/Cube/Rollup

WindowExpressions with a AggregateExpression window function with isDistinct flag on

Fails analysis with the error message:

Distinct window functions are not supported: [w]

Example:

val windowedDistinctCountExpr = "COUNT(DISTINCT 1) OVER (PARTITION BY value)"
scala> spark.emptyDataset[Int].selectExpr(windowedDistinctCountExpr)
org.apache.spark.sql.AnalysisException: Distinct window functions are not supported: count(distinct 1) windowspecdefinition(value#95, ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING);;
Project [COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L]
+- Project [value#95, COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L, COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L]
   +- Window [count(distinct 1) windowspecdefinition(value#95, ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L], [value#95]
      +- Project [value#95]
         +- LocalRelation <empty>, [value#95]

  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:40)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:90)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:108)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86)

WindowExpressions with a OffsetWindowFunction window function with an empty order specification or a non-offset window frame specification

Fails analysis with the error message:

An offset window function can only be evaluated in an ordered row-based window frame with a single offset: [windowExpr]

WindowExpressions with a window function that is not one of the following expressions: AggregateExpression, AggregateWindowFunction or OffsetWindowFunction

Fails analysis with the error message:

Expression '[e]' not supported within a window function.

Nondeterministic expressions

FIXME

UnresolvedHint

FIXME

FIXME

FIXME

After the validations, checkAnalysis executes additional check rules for correct analysis.

checkAnalysis then checks if plan is analyzed correctly (i.e. no logical plans are left unresolved). If there is one, checkAnalysis fails the analysis with AnalysisException and the following error message:

unresolved operator [o.simpleString]

In the end, checkAnalysis marks the entire logical plan as analyzed.

Note

checkAnalysis is used when:

Extended Analysis Check Rules — extendedCheckRules Extension Point

extendedCheckRules: Seq[LogicalPlan => Unit]

extendedCheckRules is a collection of rules (functions) that checkAnalysis uses for custom analysis checks (after the main validations have been executed).

Note
 When a condition of a rule does not hold the function throws an AnalysisException directly or using failAnalysis method.

checkSubqueryExpression Internal Method

checkSubqueryExpression(plan: LogicalPlan, expr: SubqueryExpression): Unit

checkSubqueryExpression…​FIXME

Note
 checkSubqueryExpression is used exclusively when CheckAnalysis is requested to validate analysis of a logical plan (for SubqueryExpression expressions).

results matching ""

    No results matching ""