StreamingSymmetricHashJoinHelper Utility

StreamingSymmetricHashJoinHelper is a Scala object with the following utility methods:

getStateWatermarkPredicates

Creating JoinStateWatermarkPredicates — `getStateWatermarkPredicates` Object Method

getStateWatermarkPredicates(
  leftAttributes: Seq[Attribute],
  rightAttributes: Seq[Attribute],
  leftKeys: Seq[Expression],
  rightKeys: Seq[Expression],
  condition: Option[Expression],
  eventTimeWatermark: Option[Long]): JoinStateWatermarkPredicates

getStateWatermarkPredicates tries to find the index of the watermark attribute among the left keys first, and if not found, the right keys.

Note	The watermark attribute is defined using Dataset.withWatermark operator.

getStateWatermarkPredicates determines the state watermark predicate for the left side of a join (for the given leftAttributes, the leftKeys and the rightAttributes).

getStateWatermarkPredicates determines the state watermark predicate for the right side of a join (for the given rightAttributes, the rightKeys and the leftAttributes).

In the end, getStateWatermarkPredicates creates a JoinStateWatermarkPredicates with the left- and right-side state watermark predicates.

Note	`getStateWatermarkPredicates` is used exclusively when `IncrementalExecution` is requested to apply the state preparation rule for batch-specific configuration (while optimizing query plans with StreamingSymmetricHashJoinExec physical operators).

Join State Watermark Predicate (for One Side of Join) — `getOneSideStateWatermarkPredicate` Internal Method

getOneSideStateWatermarkPredicate(
  oneSideInputAttributes: Seq[Attribute],
  oneSideJoinKeys: Seq[Expression],
  otherSideInputAttributes: Seq[Attribute]): Option[JoinStateWatermarkPredicate]

getOneSideStateWatermarkPredicate finds what attributes were used to define the watermark attribute (the oneSideInputAttributes attributes, the left or right join keys) and creates a JoinStateWatermarkPredicate as follows:

JoinStateKeyWatermarkPredicate if the watermark was defined on a join key (with the watermark expression for the index of the join key expression)
JoinStateValueWatermarkPredicate if the watermark was defined among the oneSideInputAttributes (with the state value watermark based on the given oneSideInputAttributes and otherSideInputAttributes)

Note	`getOneSideStateWatermarkPredicate` creates no JoinStateWatermarkPredicate (`None`) for no watermark found.

Note	`getStateWatermarkPredicates` is used exclusively to create a JoinStateWatermarkPredicates.

StreamingSymmetricHashJoinHelper