ShuffleClient — Contract to Fetch Shuffle Blocks

ShuffleClient is the contract of clients that can fetch shuffle block files.

ShuffleClient can optionally be initialized with an appId (that actually does nothing by default)

ShuffleClient has shuffle-related Spark metrics that are used when BlockManager is requested for a shuffle-related Spark metrics source (only when Executor is created for a non-local / cluster mode).

package org.apache.spark.network.shuffle;

abstract class ShuffleClient implements Closeable {
  // only required methods that have no implementation
  // the others follow
  abstract void fetchBlocks(
      String host,
      int port,
      String execId,
      String[] blockIds,
      BlockFetchingListener listener,
      TempFileManager tempFileManager);
}
Table 1. (Subset of) ShuffleClient Contract
Method Description

fetchBlocks

Fetches a sequence of blocks from a remote block manager node asynchronously

Used exclusively when ShuffleBlockFetcherIterator is requested to sendRequest

Table 2. ShuffleClients
ShuffleClient Description

BlockTransferService

ExternalShuffleClient

init Method

void init(String appId)

init does nothing by default.

Note

init is used when:

  • BlockManager is requested to initialize

  • Spark on Mesos' MesosCoarseGrainedSchedulerBackend is requested to registered

MetricSet shuffleMetrics()

shuffleMetrics returns an empty Dropwizard Metrics' MetricSet by default.

Note
shuffleMetrics is used exclusively when BlockManager is requested for a shuffle-related Spark metrics source (only when Executor is created for a non-local / cluster mode).

results matching ""

    No results matching ""