Vectorized Parquet Reader

Vectorized Parquet Reader (aka Vectorized Parquet Decoding) allows for reading datasets in parquet format in batches, i.e. rows are decoded in batches. That aims at improving memory locality and cache utilization.

The parquet encodings are largely designed to decode faster in batches, column by column. This can speed up the decoding considerably.

VectorizedParquetRecordReader is responsible for vectorized decoding and is used only when spark.sql.parquet.enableVectorizedReader configuration property is enabled and the result schema uses AtomicType data types only.

spark.sql.parquet.enableVectorizedReader Configuration Property

spark.sql.parquet.enableVectorizedReader configuration property is on by default.

val isParquetVectorizedReaderEnabled = spark.conf.get("spark.sql.parquet.enableVectorizedReader").toBoolean
assert(isParquetVectorizedReaderEnabled, "spark.sql.parquet.enableVectorizedReader should be enabled by default")

results matching ""

    No results matching ""