org.apache.spark.SparkException: Transform function must not return null. Return SparkContext.emptyRDD() instead to represent no element as the result of transformation. at org.apache.spark.streaming.dstream.TransformedDStream.compute(TransformedDStream.scala:48)
TransformedDStream
It is constructed with a collection of parents
dstreams and transformFunc
transform function.
Note
|
When created, it asserts that the input collection of dstreams use the same StreamingContext and slide interval. |
Note
|
It is acceptable to have more than one dependent dstream. |
The dependencies is the input collection of dstreams.
The slide interval is exactly the same as that in the first dstream in parents
.
When requested to compute a RDD, it goes over every dstream in parents
and asks to getOrCompute a RDD.
Note
|
It may throw a SparkException when a dstream does not compute a RDD for a batch.
|
Caution
|
FIXME Prepare an example to face the exception. |
It then calls transformFunc
with the collection of RDDs.
If the transform function returns null
a SparkException is thrown:
The result of transformFunc
is returned.