MemoryStreamBase Contract — Base Contract for Memory Sources

MemoryStreamBase is the base of the BaseStreamingSource contract for memory sources that can add data.

Table 1. MemoryStreamBase Contract
Method Description

addData

 
addData(
  data: TraversableOnce[A]): Offset

logicalPlan

 
logicalPlan: LogicalPlan
Table 2. MemoryStreamBases
MemoryStreamBase Description

ContinuousMemoryStream

MemoryStream

MicroBatchReader for Micro-Batch Stream Processing

Creating MemoryStreamBase Instance

MemoryStreamBase takes the following to be created:

  • SQLContext

Note
 MemoryStreamBase is a Scala abstract class and cannot be created directly. It is created indirectly for the concrete MemoryStreamBases.

Creating Streaming Dataset — toDS Method

toDS(): Dataset[A]

toDS simply creates a Dataset (for the sqlContext and the logicalPlan)

Creating Streaming DataFrame — toDF Method

toDF(): DataFrame

toDF simply creates a Dataset of rows (for the sqlContext and the logicalPlan)

Internal Properties

Name Description

attributes

Schema attributes of the encoder (Seq[AttributeReference])

Used when…​FIXME

encoder

Spark SQL’s ExpressionEncoder for the data

Used when…​FIXME

results matching ""

    No results matching ""