SparkSqlParser — Default SQL Parser

SparkSqlParser is the default SQL parser of the SQL statements supported in Spark SQL.

SparkSqlParser supports variable substitution.

SparkSqlParser uses SparkSqlAstBuilder (as AstBuilder).

Spark SQL supports SQL statements as described in SqlBase.g4 ANTLR grammar.

SparkSqlParser is available as sqlParser of a SessionState.

val spark: SparkSession = ...

SparkSqlParser is used to translate an expression to the corresponding Column in the following:

scala> expr("token = 'hello'")
16/07/07 18:32:53 INFO SparkSqlParser: Parsing command: token = 'hello'
res0: org.apache.spark.sql.Column = (token = hello)

SparkSqlParser is used to parse table strings into their corresponding table identifiers in the following:

SparkSqlParser is used to translate a SQL text to its corresponding logical operator in SparkSession.sql method.


Enable INFO logging level for org.apache.spark.sql.execution.SparkSqlParser logger to see what happens inside.

Add the following line to conf/

Refer to Logging.

Variable Substitution

FIXME See SparkSqlParser and substitutor.

results matching ""

    No results matching ""