SaveAs Operators

There are two saveAs operators in DStream:

  • saveAsObjectFiles

  • saveAsTextFiles

They are output operators that return nothing as they save each RDD in a batch to a storage.

Their full signature is as follows:

saveAsObjectFiles(prefix: String, suffix: String = ""): Unit
saveAsTextFiles(prefix: String, suffix: String = ""): Unit
Note
 SaveAs operators use foreachRDD output operator.

saveAsObjectFiles uses RDD.saveAsObjectFile while saveAsTextFiles uses RDD.saveAsTextFile.

The file name is based on mandatory prefix and batch time with optional suffix. It is in the format of [prefix]-[time in milliseconds].[suffix].

Example

val clicks: InputDStream[(String, String)] = messages
clicks.saveAsTextFiles("clicks", "txt")

results matching ""

    No results matching ""