Data Types

DataType abstract class is the base type of all built-in data types in Spark SQL, e.g. strings, longs.

DataType has two main type families:

  • Atomic Types as an internal type to represent types that are not null, UDTs, arrays, structs, and maps

  • Numeric Types with fractional and integral types

Table 1. Standard Data Types
Type Family Data Type Scala Types

Atomic Types

(except fractional and integral types)

BinaryType

BooleanType

DateType

StringType

TimestampType

java.sql.Timestamp

Fractional Types

(concrete NumericType)

DecimalType

DoubleType

FloatType

Integral Types

(concrete NumericType)

ByteType

IntegerType

LongType

ShortType

ArrayType

CalendarIntervalType

MapType

NullType

ObjectType

StructType

UserDefinedType

AnyDataType

Matches any concrete data type
Caution
 FIXME What about AbstractDataType?

You can extend the type system and create your own user-defined types (UDTs).

The DataType Contract defines methods to build SQL, JSON and string representations.

Note
 DataType (and the concrete Spark SQL types) live in org.apache.spark.sql.types package. 
import org.apache.spark.sql.types.StringType

scala> StringType.json
res0: String = "string"

scala> StringType.sql
res1: String = STRING

scala> StringType.catalogString
res2: String = string

You should use DataTypes object in your code to create complex Spark SQL types, i.e. arrays or maps.

import org.apache.spark.sql.types.DataTypes

scala> val arrayType = DataTypes.createArrayType(BooleanType)
arrayType: org.apache.spark.sql.types.ArrayType = ArrayType(BooleanType,true)

scala> val mapType = DataTypes.createMapType(StringType, LongType)
mapType: org.apache.spark.sql.types.MapType = MapType(StringType,LongType,true)

DataType has support for Scala’s pattern matching using unapply method.

???

DataType Contract

Any type in Spark SQL follows the DataType contract which means that the types define the following methods:

  • json and prettyJson to build JSON representations of a data type

  • defaultSize to know the default size of values of a type

  • simpleString and catalogString to build user-friendly string representations (with the latter for external catalogs)

  • sql to build SQL representation

import org.apache.spark.sql.types.DataTypes._

val maps = StructType(
  StructField("longs2strings", createMapType(LongType, StringType), false) :: Nil)

scala> maps.prettyJson
res0: String =
{
  "type" : "struct",
  "fields" : [ {
    "name" : "longs2strings",
    "type" : {
      "type" : "map",
      "keyType" : "long",
      "valueType" : "string",
      "valueContainsNull" : true
    },
    "nullable" : false,
    "metadata" : { }
  } ]
}

scala> maps.defaultSize
res1: Int = 2800

scala> maps.simpleString
res2: String = struct<longs2strings:map<bigint,string>>

scala> maps.catalogString
res3: String = struct<longs2strings:map<bigint,string>>

scala> maps.sql
res4: String = STRUCT<`longs2strings`: MAP<BIGINT, STRING>>

DataTypes — Factory Methods for Data Types

DataTypes is a Java class with methods to access simple or create complex DataType types in Spark SQL, i.e. arrays and maps.

Tip
 It is recommended to use DataTypes class to define DataType types in a schema.

DataTypes lives in org.apache.spark.sql.types package.

import org.apache.spark.sql.types.DataTypes

scala> val arrayType = DataTypes.createArrayType(BooleanType)
arrayType: org.apache.spark.sql.types.ArrayType = ArrayType(BooleanType,true)

scala> val mapType = DataTypes.createMapType(StringType, LongType)
mapType: org.apache.spark.sql.types.MapType = MapType(StringType,LongType,true)
Note

Simple DataType types themselves, i.e. StringType or CalendarIntervalType, come with their own Scala’s case objects alongside their definitions.

You may also import the types package and have access to the types.

import org.apache.spark.sql.types._

UDTs — User-Defined Types

Caution
 FIXME

Converting DDL into DataType — fromDDL Utility

fromDDL(ddl: String): DataType

fromDDL…​FIXME

Note
 fromDDL is used when…​FIXME

results matching ""

    No results matching ""