MemoryManager Configuration Properties

The page describes the memory-related configuration properties of Apache Spark (mostly MemoryManager and "relatives").

Note
It may be that the page should be somewhere else. Any suggestions?

Use the following idiom to set the configuration properties.

import org.apache.spark.internal.config._
import org.apache.spark.SparkConf
// all properties are private[spark], sorry
// the following won't work unless the code is under org.apache.spark package
val sparkConf = new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false")

import org.apache.spark.SparkConf
val sparkConf = new SparkConf()
scala> println(sparkConf.getOption("spark.memory.offHeap.enabled"))
None

// Using ConfigEntry-based SparkConf.set that is also private[spark]
// Use :pa -raw in spark-shell to paste the following snippet
// BEGIN
package org.apache.spark.japila

object Config {
  import org.apache.spark.SparkConf
  import org.apache.spark.internal.config._
  def with_MEMORY_OFFHEAP_ENABLED(sparkConf: SparkConf = new SparkConf()): SparkConf = {
    sparkConf.set(MEMORY_OFFHEAP_ENABLED, true)
  }
}
// END

import org.apache.spark.japila.Config
val sparkConf_OffHeap = Config.with_MEMORY_OFFHEAP_ENABLED(sparkConf)

scala> println(sparkConf_OffHeap.get("spark.memory.offHeap.enabled"))
true
Table 1. Memory-Related Configuration Properties
Name Default Value Description

spark.memory.offHeap.enabled

false

Controls whether Spark will attempt to use off-heap memory for certain operations (true) or not (false).

Tracks whether Tungsten memory will be allocated on the JVM heap or off-heap (using sun.misc.Unsafe).

If enabled, spark.memory.offHeap.size has to be greater than 0.

Used exclusively when MemoryManager is requested for tungstenMemoryMode.

Note
Before Apache Spark 1.6, spark.memory.offHeap.enabled was known as spark.unsafe.offHeap.

spark.memory.offHeap.size

results matching ""

    No results matching ""