Scheduling Mode — spark.scheduler.mode Spark Property

Scheduling Mode (aka order task policy or scheduling policy or scheduling order) defines a policy to sort tasks in order for execution.

The scheduling mode schedulingMode attribute is part of the TaskScheduler Contract.

The only implementation of the TaskScheduler contract in Spark — TaskSchedulerImpl — uses spark.scheduler.mode setting to configure schedulingMode that is merely used to set up the rootPool attribute (with FIFO being the default). It happens when TaskSchedulerImpl is initialized.

There are three acceptable scheduling modes:

Note
Out of three possible SchedulingMode policies only FIFO and FAIR modes are supported by TaskSchedulerImpl.
Note

After the root pool is initialized, the scheduling mode is no longer relevant (since the Schedulable that represents the root pool is fully set up).

Note
The root pool is a Schedulable. Refer to Schedulable.

Monitoring FAIR Scheduling Mode using Spark UI

Caution
FIXME Describe me…​

results matching ""

    No results matching ""