ReplayListenerBus is a custom SparkListenerBus that can replay JSON-encoded SparkListenerEvent events.

ReplayListenerBus is used in FsHistoryProvider.
ReplayListenerBus is a private[spark] class in org.apache.spark.scheduler package.

Replaying JSON-encoded SparkListenerEvents from Stream — replay Method

  logData: InputStream,
  sourceName: String,
  maybeTruncated: Boolean = false): Unit

replay reads JSON-encoded SparkListenerEvent events from logData (one event per line) and posts them to all registered SparkListenerInterface listeners.

replay uses jackson from json4s library to parse the AST for JSON.

When there is an exception parsing a JSON event, you may see the following WARN message in the logs (for the last line) or a JsonParseException.

WARN Got JsonParseException from log file $sourceName at line [lineNumber], the file might not have finished writing cleanly.

Any other non-IO exceptions end up with the following ERROR messages in the logs:

ERROR Exception parsing Spark event log: [sourceName]
ERROR Malformed line #[lineNumber]: [currentLine]
The sourceName input argument is only used for messages.

results matching ""

    No results matching ""