The Internals of Spark Structured Streaming
Introduction
Spark Structured Streaming and Streaming Queries
- Batch Processing Time
Internals of Streaming Queries
Streaming Join
Streaming Join
StateStoreAwareZipPartitionsRDD
SymmetricHashJoinStateManager
OneSideHashJoiner
JoinStateWatermarkPredicates
- JoinStateWatermarkPredicate
StateStoreAwareZipPartitionsHelper
StreamingSymmetricHashJoinHelper
StreamingJoinHelper
Extending Structured Streaming with New Data Sources
Extending Structured Streaming with New Data Sources
BaseStreamingSource
BaseStreamingSink
StreamWriteSupport
StreamWriter
DataSource
Demos
Demos
Internals of FlatMapGroupsWithStateExec Physical Operator
Arbitrary Stateful Streaming Aggregation with KeyValueGroupedDataset.flatMapGroupsWithState Operator
Exploring Checkpointed State
Streaming Watermark with Aggregation in Append Output Mode
Streaming Query for Running Counts (Socket Source and Complete Output Mode)
Streaming Aggregation with Kafka Data Source
groupByKey Streaming Aggregation in Update Mode
StateStoreSaveExec with Complete Output Mode
StateStoreSaveExec with Update Output Mode
Developing Custom Streaming Sink (and Monitoring SQL Queries in web UI)
current_timestamp Function For Processing Time in Streaming Queries
Using StreamingQueryManager for Query Termination Management
Streaming Aggregation
Streaming Aggregation
StateStoreRDD
- StateStoreOps
StreamingAggregationStateManager
Stateful Stream Processing
Stateful Stream Processing
Streaming Watermark
Streaming Deduplication
Streaming Limit
StateStore
- StateStoreId
- HDFSBackedStateStore
StateStoreProvider
- StateStoreProviderId
- HDFSBackedStateStoreProvider
StateStoreCoordinator
- StateStoreCoordinatorRef
WatermarkSupport
StatefulOperator
- StateStoreReader
- StateStoreWriter
StatefulOperatorStateInfo
StateStoreMetrics
StateStoreCustomMetric
StateStoreUpdater
EventTimeStatsAccum
StateStoreConf
Arbitrary Stateful Streaming Aggregation
Arbitrary Stateful Streaming Aggregation
GroupState
- GroupStateImpl
GroupStateTimeout
StateManager
FlatMapGroupsWithStateExecHelper Helper Class
InputProcessor Helper Class of FlatMapGroupsWithStateExec Physical Operator
Developing Streaming Applications
DataStreamReader
DataStreamWriter
- OutputMode
- Trigger
StreamingQuery
Streaming Operators
window Function
KeyValueGroupedDataset
- mapGroupsWithState Operator
- flatMapGroupsWithState Operator
StreamingQueryManager
SQLConf
Configuration Properties
Monitoring of Streaming Query Execution
StreamingQueryListener
ProgressReporter
StreamingQueryProgress
StreamingQueryStatus
MetricsReporter
Web UI
Logging
File-Based Data Source
FileStreamSource
FileStreamSink
FileStreamSinkLog
SinkFileStatus
ManifestFileCommitProtocol
MetadataLogFileIndex
Kafka Data Source
Kafka Data Source
KafkaSourceProvider
KafkaSource
KafkaRelation
KafkaSourceRDD
CachedKafkaConsumer
KafkaSourceOffset
KafkaOffsetReader
ConsumerStrategy
KafkaSink
KafkaOffsetRangeLimit
KafkaDataConsumer
KafkaMicroBatchReader
KafkaContinuousReader
- KafkaContinuousInputPartition
Text Socket Data Source
TextSocketSourceProvider
TextSocketSource
Rate Data Source
RateSourceProvider
RateStreamSource
RateStreamMicroBatchReader
Console Data Sink
ConsoleSinkProvider
ConsoleWriter
Foreach Data Sink
ForeachWriterProvider
ForeachWriter
ForeachSink
ForeachBatch Data Sink
ForeachBatchSink
Memory Data Source
Memory Data Source
MemoryStream
ContinuousMemoryStream
MemorySink
MemorySinkV2
- MemoryStreamWriter
MemoryStreamBase
MemorySinkBase
Offsets and Metadata Checkpointing (Fault-Tolerance and Reliability)
Offsets and Metadata Checkpointing
MetadataLog
HDFSMetadataLog
CommitLog
- CommitMetadata
OffsetSeqLog
- OffsetSeq
CompactibleFileStreamLog
- FileStreamSourceLog
OffsetSeqMetadata
CheckpointFileManager
- FileContextBasedCheckpointFileManager
- FileSystemBasedCheckpointFileManager
Offset
StreamProgress
Micro-Batch Stream Processing (Structured Streaming V1)
Micro-Batch Stream Processing
MicroBatchExecution
- MicroBatchWriter
MicroBatchReadSupport
- MicroBatchReader
WatermarkTracker
Source
- StreamSourceProvider
Sink
- StreamSinkProvider
Continuous Stream Processing (Structured Streaming V2)
Continuous Stream Processing
ContinuousExecution
ContinuousReadSupport Contract
ContinuousReader Contract
RateStreamContinuousReader
EpochCoordinator RPC Endpoint
- EpochCoordinatorRef
- EpochTracker
ContinuousQueuedDataReader
- DataReaderThread
- EpochMarkerGenerator
PartitionOffset
ContinuousExecutionRelation Leaf Logical Operator
WriteToContinuousDataSource Unary Logical Operator
WriteToContinuousDataSourceExec Unary Physical Operator
- ContinuousWriteRDD
ContinuousDataSourceRDD
Query Planning and Execution
StreamExecution
- StreamingQueryWrapper
TriggerExecutor
IncrementalExecution
StreamingQueryListenerBus
StreamMetadata
Logical Operators
EventTimeWatermark Unary Logical Operator
FlatMapGroupsWithState Unary Logical Operator
Deduplicate Unary Logical Operator
MemoryPlan Logical Query Plan
StreamingRelation Leaf Logical Operator for Streaming Source
StreamingRelationV2 Leaf Logical Operator
StreamingExecutionRelation Leaf Logical Operator for Streaming Source At Execution
Physical Operators
EventTimeWatermarkExec
FlatMapGroupsWithStateExec
StateStoreRestoreExec
StateStoreSaveExec
StreamingDeduplicateExec
StreamingGlobalLimitExec
StreamingRelationExec
StreamingSymmetricHashJoinExec
Execution Planning Strategies
FlatMapGroupsWithStateStrategy
StatefulAggregationStrategy
StreamingDeduplicationStrategy
StreamingGlobalLimitStrategy
StreamingJoinStrategy
StreamingRelationStrategy
Varia
UnsupportedOperationChecker

Powered by GitBook

SinkProgress

SinkProgress

SinkProgress is…FIXME

results matching ""

No results matching ""