|FIXME Picture it.|
It uses the reference to the remote ApplicationMaster RPC Endpoint to send messages to.
Add the following line to
Refer to Logging.
RequestExecutors( requestedTotal: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]) extends CoarseGrainedClusterMessage
RequestExecutors is to inform ApplicationMaster about the current requirements for the total number of executors (as
requestedTotal), including already pending and running executors.
Any issues communicating with the remote
ApplicationMaster RPC endpoint are reported as ERROR messages in the logs:
ERROR Sending RequestExecutors to AM was unsuccessful
AddWebUIFilter( filterName: String, filterParams: Map[String, String], proxyBase: String)
AddWebUIFilter triggers setting
spark.ui.proxyBase system property and adding the
filterName filter to web UI.
AddWebUIFilter is sent by
ApplicationMaster when it adds
AmIpFilter to web UI.
It firstly sets
spark.ui.proxyBase system property to the input
proxyBase (if not empty).
If it defines a filter, i.e. the input
filterParams are both not empty, you should see the following INFO message in the logs:
INFO Add WebUI Filter. [filterName], [filterParams], [proxyBase]
It then sets
spark.ui.filters to be the input
filterName in the internal
conf SparkConf attribute.
filterParams are also set as
The filter is added to web UI using
RegisterClusterManager message arrives, the following INFO message is printed out to the logs:
INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as [am]
The internal reference to the remote ApplicationMaster RPC Endpoint is set (to
RetrieveLastAllocatedExecutorId is received,
YarnSchedulerEndpoint responds with the current value of currentExecutorIdCounter.
onDisconnected clears the internal reference to the remote ApplicationMaster RPC Endpoint (i.e. it sets it to
None) if the remote address matches the reference’s.
|It is a callback method to be called when…FIXME|
You should see the following WARN message in the logs if that happens:
WARN ApplicationMaster has disassociated: [remoteAddress]
onStop shuts askAmThreadPool down immediately.
amEndpoint is a reference to a remote ApplicationMaster RPC Endpoint.