StructField — Single Field in StructType

StructField describes a single field in a StructType with the following:

  • Name

  • DataType

  • nullable flag (enabled by default)

  • Metadata (empty by default)

A comment is part of metadata under comment key and is used to build a Hive column or when describing a table.

scala> schemaTyped("a").getComment
res0: Option[String] = None

scala> schemaTyped("a").withComment("this is a comment").getComment
res1: Option[String] = Some(this is a comment)

As of Spark 2.4.0, StructField can be converted to DDL format using toDDL method.

Example: Using StructField.toDDL
import org.apache.spark.sql.types.MetadataBuilder
val metadata = new MetadataBuilder()
  .putString("comment", "this is a comment")
import org.apache.spark.sql.types.{LongType, StructField}
val f = new StructField(name = "id", dataType = LongType, nullable = false, metadata)
scala> println(f.toDDL)
`id` BIGINT COMMENT 'this is a comment'

Converting to DDL Format — toDDL Method

toDDL: String

toDDL gives a text in the format:

[quoted name] [dataType][optional comment]

toDDL is used when:

results matching ""

    No results matching ""