HashedRelation is the contract for "relations" with values hashed by some key.

HashedRelation is a KnownSizeEstimation.

package org.apache.spark.sql.execution.joins

trait HashedRelation extends KnownSizeEstimation {
  // only required methods that have no implementation
  // the others follow
  def asReadOnlyCopy(): HashedRelation
  def close(): Unit
  def get(key: InternalRow): Iterator[InternalRow]
  def getAverageProbesPerLookup: Double
  def getValue(key: InternalRow): InternalRow
  def keyIsUnique: Boolean
HashedRelation is a private[execution] contract.
Table 1. HashedRelation Contract
Method Description


Gives a read-only copy of this HashedRelation to be safely used in a separate thread.

Used exclusively when BroadcastHashJoinExec is requested to execute (and transform every partitions of streamedPlan physical operator using the broadcast variable of buildPlan physical operator).


Gives internal rows for the given key or null

Used when HashJoin is requested to innerJoin, outerJoin, semiJoin, existenceJoin and antiJoin.


Gives the value internal row for a given key

HashedRelation has two variants of getValue, i.e. one that accepts an InternalRow and another a Long. getValue with an InternalRow does not seem to be used at all.


Used when…​FIXME

getValue Method

getValue(key: Long): InternalRow
This is getValue that takes a long key. There is the more generic getValue that takes an internal row instead.

getValue simply reports an UnsupportedOperationException (and expects concrete HashedRelations to provide a more meaningful implementation).

getValue is used exclusively when LongHashedRelation is requested to get the value for a given key.

Creating Concrete HashedRelation Instance (for Build Side of Hash-based Join) — apply Factory Method

  input: Iterator[InternalRow],
  key: Seq[Expression],
  sizeEstimate: Int = 64,
  taskMemoryManager: TaskMemoryManager = null): HashedRelation

apply creates a LongHashedRelation when the input key collection has a single expression of type long or UnsafeHashedRelation otherwise.


The input key expressions are:


apply is used when:

results matching ""

    No results matching ""