JsonToStructs Unary Expression

JsonToStructs is a unary expression with timezone support and CodegenFallback.

JsonToStructs is created to represent from_json function.

import org.apache.spark.sql.functions.from_json
val jsonCol = from_json($"json", new StructType())

import org.apache.spark.sql.catalyst.expressions.JsonToStructs
val jsonExpr = jsonCol.expr.asInstanceOf[JsonToStructs]
scala> println(jsonExpr.numberedTreeString)
00 jsontostructs('json, None)
01 +- 'json

JsonToStructs is a ExpectsInputTypes expression.

Note

JsonToStructs uses JacksonParser in FAILFAST mode that simply fails early when a corrupted/malformed record is found (and hence does not support columnNameOfCorruptRecord JSON option).
Table 1. JsonToStructs’s Properties
Property Description

converter

Function that converts Seq[InternalRow] into…​FIXME

nullable

Enabled (i.e. true)

parser

JacksonParser with rowSchema and JSON options

Note
 JSON options are made up of the input options with mode option as FAILFAST and the input time zone as the default time zone.

rowSchema

StructType that…​FIXME

  • schema when of type StructType

  • StructType of the elements in schema when of type ArrayType

Creating JsonToStructs Instance

JsonToStructs takes the following when created:

JsonToStructs initializes the internal registries and counters.

Parsing Table Schema for String Literals — validateSchemaLiteral Method

validateSchemaLiteral(exp: Expression): StructType

validateSchemaLiteral requests CatalystSqlParser to parseTableSchema for Literal of StringType.

For any other non-StringType types, validateSchemaLiteral reports a AnalysisException:

Expected a string literal instead of [expression]

results matching ""

    No results matching ""