CompressionCodecs

CompressionCodecs is a utility object…​FIXME

Table 1. Known Compression Codecs
Alias Fully-Qualified Class Name

none

uncompressed

bzip2

org.apache.hadoop.io.compress.BZip2Codec

deflate

org.apache.hadoop.io.compress.DeflateCodec

gzip

org.apache.hadoop.io.compress.GzipCodec

lz4

org.apache.hadoop.io.compress.Lz4Codec

snappy

org.apache.hadoop.io.compress.SnappyCodec

setCodecConfiguration Method

setCodecConfiguration(conf: Configuration, codec: String): Unit

setCodecConfiguration sets compression-related configurations to the Hadoop Configuration per the input codec.

Note
The input codec should be a fully-qualified class name, i.e. org.apache.hadoop.io.compress.SnappyCodec.

If the input codec is defined (i.e. not null), setCodecConfiguration sets the following configuration properties.

Table 2. Compression-Related Hadoop Configuration Properties (codec defined)
Name Value

mapreduce.output.fileoutputformat.compress

true

mapreduce.output.fileoutputformat.compress.type

BLOCK

mapreduce.output.fileoutputformat.compress.codec

The input codec name

mapreduce.map.output.compress

true

mapreduce.map.output.compress.codec

The input codec name

If the input codec is not defined (i.e. null), setCodecConfiguration sets the following configuration properties.

Table 3. Compression-Related Hadoop Configuration Properties (codec not defined)
Name Value

mapreduce.output.fileoutputformat.compress

false

mapreduce.map.output.compress

false

Note
setCodecConfiguration is used when CSVFileFormat, JsonFileFormat and TextFileFormat are requested to prepareWrite.

results matching ""

    No results matching ""