AvroOptions — Avro Data Source Options

AvroOptions represents the options of the Avro data source.

Table 1. Options for Avro Data Source
Option / Key Default Value Description

avroSchema

(undefined)

Avro schema in JSON format

compression

(undefined)

Specifies the compression codec to use when writing Avro data to disk

Note
 If the option is not defined explicitly, Avro data source uses spark.sql.avro.compression.codec configuration property.

ignoreExtension

false

Controls whether Avro data source should read all Avro files regardless of their extension (true) or not (false)

By default, Avro data source reads only files with .avro file extension.

Note
 If the option is not defined explicitly, Avro data source uses avro.mapred.ignore.inputs.without.extension Hadoop runtime property.

recordName

topLevelRecord

Top-level record name when writing Avro data to disk

Consult Apache Avro™ 1.8.2 Specification

recordNamespace

(empty)

Record namespace when writing Avro data to disk

Consult Apache Avro™ 1.8.2 Specification
Note
 The options are case-insensitive.

AvroOptions is created when AvroFileFormat is requested to inferSchema, prepareWrite and buildReader.

Creating AvroOptions Instance

AvroOptions takes the following when created:

  • Case-insensitive configuration parameters (i.e. Map[String, String])

  • Hadoop Configuration

results matching ""

    No results matching ""