StreamingJoinStrategy Execution Planning Strategy — Stream-Stream Equi-Joins

StreamingJoinStrategy is an execution planning strategy that can plan streaming queries with Join logical operators of two streaming queries to a StreamingSymmetricHashJoinExec physical operator.

Tip
 Read up on Execution Planning Strategies in The Internals of Spark SQL online book.

StreamingJoinStrategy throws an AnalysisException when applied to a Join logical operator with no equality predicate:

Stream-stream join without equality predicate is not supported

StreamingJoinStrategy is used exclusively when IncrementalExecution is requested to plan a streaming query.

Tip

StreamingJoinStrategy does not print out any messages to the logs. StreamingJoinStrategy however uses ExtractEquiJoinKeys Scala extractor for destructuring Join logical operators that does print out DEBUG messages to the logs.

Read up on ExtractEquiJoinKeys in The Internals of Spark SQL online book.

Enable ALL logging level for org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys=ALL

Refer to Logging.

results matching ""

    No results matching ""