Transformations — Lazy Operations on RDD (to Create One or More RDDs)

Transformations are lazy operations on an RDD that create one or many new RDDs.

// T and U are Scala types
transformation: RDD[T] => RDD[U]
transformation: RDD[T] => Seq[RDD[U]]

In other words, transformations are functions that take an RDD as the input and produce one or many RDDs as the output. Transformations do not change the input RDD (since RDDs are immutable and hence cannot be modified), but produce one or more new RDDs by applying the computations they represent.

Table 1. (Subset of) RDD Transformations (Public API)
Method Description

aggregate

aggregate[U](zeroValue: U)(
  seqOp:  (U, T) => U,
  combOp: (U, U) => U): U

barrier

barrier(): RDDBarrier[T]

(New in 2.4.0) Marks the current stage as a barrier stage in Barrier Execution Mode, where Spark must launch all tasks together

Internally, barrier creates a RDDBarrier over the RDD

cache

cache(): this.type

Persists the RDD with the MEMORY_ONLY storage level

Synonym of persist

cache

cache(): this.type

Synonym of persist

filter

filter(f: T => Boolean): RDD[T]

flatMap

flatMap[U](f: T => TraversableOnce[U]): RDD[U]

map

map[U](f: T => U): RDD[U]

mapPartitions

mapPartitions[U](
  f: Iterator[T] => Iterator[U],
  preservesPartitioning: Boolean = false): RDD[U]

mapPartitionsWithIndex

mapPartitionsWithIndex[U](
  f: (Int, Iterator[T]) => Iterator[U],
  preservesPartitioning: Boolean = false): RDD[U]

randomSplit

randomSplit(
  weights: Array[Double],
  seed: Long = Utils.random.nextLong): Array[RDD[T]]

union

++(other: RDD[T]): RDD[T]
union(other: RDD[T]): RDD[T]

persist

persist(): this.type
persist(newLevel: StorageLevel): this.type

By applying transformations you incrementally build a RDD lineage with all the parent RDDs of the final RDD(s).

Transformations are lazy, i.e. are not executed immediately. Only after calling an action are transformations executed.

After executing a transformation, the result RDD(s) will always be different from their parents and can be smaller (e.g. filter, count, distinct, sample), bigger (e.g. flatMap, union, cartesian) or the same size (e.g. map).

Caution
There are transformations that may trigger jobs, e.g. sortBy, zipWithIndex, etc.
rdd sparkcontext transformations action.png
Figure 1. From SparkContext by transformations to the result

Certain transformations can be pipelined which is an optimization that Spark uses to improve performance of computations.

scala> val file = sc.textFile("README.md")
file: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[54] at textFile at <console>:24

scala> val allWords = file.flatMap(_.split("\\W+"))
allWords: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[55] at flatMap at <console>:26

scala> val words = allWords.filter(!_.isEmpty)
words: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[56] at filter at <console>:28

scala> val pairs = words.map((_,1))
pairs: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[57] at map at <console>:30

scala> val reducedByKey = pairs.reduceByKey(_ + _)
reducedByKey: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[59] at reduceByKey at <console>:32

scala> val top10words = reducedByKey.takeOrdered(10)(Ordering[Int].reverse.on(_._2))
INFO SparkContext: Starting job: takeOrdered at <console>:34
...
INFO DAGScheduler: Job 18 finished: takeOrdered at <console>:34, took 0.074386 s
top10words: Array[(String, Int)] = Array((the,21), (to,14), (Spark,13), (for,11), (and,10), (##,8), (a,8), (run,7), (can,6), (is,6))

There are two kinds of transformations:

Narrow Transformations

Narrow transformations are the result of map, filter and such that is from the data from a single partition only, i.e. it is self-sustained.

An output RDD has partitions with records that originate from a single partition in the parent RDD. Only a limited subset of partitions used to calculate the result.

Spark groups narrow transformations as a stage which is called pipelining.

Wide Transformations

Wide transformations are the result of groupByKey and reduceByKey. The data required to compute the records in a single partition may reside in many partitions of the parent RDD.

Note
Wide transformations are also called shuffle transformations as they may or may not depend on a shuffle.

All of the tuples with the same key must end up in the same partition, processed by the same task. To satisfy these operations, Spark must execute RDD shuffle, which transfers data across cluster and results in a new stage with a new set of partitions.

zipWithIndex

zipWithIndex(): RDD[(T, Long)]

zipWithIndex zips this RDD[T] with its element indices.

Caution

If the number of partitions of the source RDD is greater than 1, it will submit an additional job to calculate start indices.

val onePartition = sc.parallelize(0 to 9, 1)

scala> onePartition.partitions.length
res0: Int = 1

// no job submitted
onePartition.zipWithIndex

val eightPartitions = sc.parallelize(0 to 9, 8)

scala> eightPartitions.partitions.length
res1: Int = 8

// submits a job
eightPartitions.zipWithIndex
spark transformations zipWithIndex webui.png
Figure 2. Spark job submitted by zipWithIndex transformation

results matching ""

    No results matching ""