SparkSqlParser — Default SQL Parser

SparkSqlParser is the default SQL parser of the SQL statements supported in Spark SQL.

SparkSqlParser supports variable substitution.

SparkSqlParser uses SparkSqlAstBuilder (as AstBuilder).

Note
Spark SQL supports SQL statements as described in SqlBase.g4 ANTLR grammar.

SparkSqlParser is available as sqlParser of a SessionState.

val spark: SparkSession = ...
spark.sessionState.sqlParser

SparkSqlParser is used to translate an expression to the corresponding Column in the following:

scala> expr("token = 'hello'")
16/07/07 18:32:53 INFO SparkSqlParser: Parsing command: token = 'hello'
res0: org.apache.spark.sql.Column = (token = hello)

SparkSqlParser is used to parse table strings into their corresponding table identifiers in the following:

SparkSqlParser is used to translate a SQL text to its corresponding logical operator in SparkSession.sql method.

Tip

Enable INFO logging level for org.apache.spark.sql.execution.SparkSqlParser logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.execution.SparkSqlParser=INFO

Refer to Logging.

Variable Substitution

Caution
FIXME See SparkSqlParser and substitutor.

results matching ""

    No results matching ""