SparkSqlParser — Default SQL Parser

SparkSqlParser is the default SQL parser of the SQL statements supported in Spark SQL.

SparkSqlParser supports variable substitution.

SparkSqlParser uses SparkSqlAstBuilder (as AstBuilder).

Note	Spark SQL supports SQL statements as described in SqlBase.g4 ANTLR grammar.

SparkSqlParser is available as sqlParser of a SessionState.

val spark: SparkSession = ...
spark.sessionState.sqlParser

SparkSqlParser is used to translate an expression to the corresponding Column in the following:

expr function
Dataset.selectExpr operator
Dataset.filter operator
Dataset.where operator

scala> expr("token = 'hello'")
16/07/07 18:32:53 INFO SparkSqlParser: Parsing command: token = 'hello'
res0: org.apache.spark.sql.Column = (token = hello)

SparkSqlParser is used to parse table strings into their corresponding table identifiers in the following:

table methods in DataFrameReader and SparkSession
insertInto and saveAsTable methods of DataFrameWriter
createExternalTable and refreshTable methods of Catalog (and SessionState)

SparkSqlParser is used to translate a SQL text to its corresponding logical operator in SparkSession.sql method.

Tip	Enable `INFO` logging level for `org.apache.spark.sql.execution.SparkSqlParser` logger to see what happens inside. Add the following line to `conf/log4j.properties`: `log4j.logger.org.apache.spark.sql.execution.SparkSqlParser=INFO` Refer to Logging.

Variable Substitution

Caution

FIXME See SparkSqlParser and substitutor.

SparkSqlParser

SparkSqlParser — Default SQL Parser

Variable Substitution

results matching ""

No results matching ""