Jobs Tab

The Jobs Tab shows status of all Spark jobs in a Spark application (i.e. a SparkContext).

spark webui jobs.png
Figure 1. Jobs Tab

The Jobs tab is available under /jobs URL, i.e. http://localhost:4040/jobs.

spark webui jobs event timeline.png
Figure 2. Event Timeline in Jobs Tab

The Jobs tab consists of two pages, i.e. All Jobs and Details for Job pages.

Internally, the Jobs Tab is represented by JobsTab class that is a custom SparkUITab with jobs prefix.

The Jobs tab uses JobProgressListener to access statistics of job executions in a Spark application to display.

Showing All Jobs — AllJobsPage Page

AllJobsPage is a page (in Jobs tab) that renders a summary, an event timeline, and active, completed, and failed jobs of a Spark application.

Jobs (in any state) are displayed when their number is greater than 0.

AllJobsPage displays the Summary section with the current Spark user, total uptime, scheduling mode, and the number of jobs per status.

AllJobsPage uses JobProgressListener for Scheduling Mode.
spark webui jobs summary section.png
Figure 3. Summary Section in Jobs Tab

Under the summary section is the Event Timeline section.

spark webui jobs event timeline.png
Figure 4. Event Timeline in Jobs Tab
AllJobsPage uses ExecutorsListener to build the event timeline.

Active Jobs, Completed Jobs, and Failed Jobs sections follow.

spark webui jobs status section.png
Figure 5. Job Status Section in Jobs Tab

Jobs are clickable, i.e. you can click on a job to see information about the stages of tasks inside it.

When you hover over a job in Event Timeline not only you see the job legend but also the job is highlighted in the Summary section.

spark webui jobs timeline popup.png
Figure 6. Hovering Over Job in Event Timeline Highlights The Job in Status Section

The Event Timeline section shows not only jobs but also executors.

spark webui jobs timeline executors.png
Figure 7. Executors in Event Timeline
Use Programmable Dynamic Allocation (using SparkContext) to manage executors for demo purposes.

Details for Job — JobPage Page

When you click a job in AllJobsPage page, you see the Details for Job page.

spark webui jobs details for job.png
Figure 8. Details for Job Page

JobPage is a custom WebUIPage that shows statistics and stage list for a given job.

Details for Job page is registered under /job URL, i.e. http://localhost:4040/jobs/job/?id=0 and accepts one mandatory id request parameter as a job identifier.

When a job id is not found, you should see "No information to display for job ID" message.

spark webui jobs details for job no job.png
Figure 9. "No information to display for job" in Details for Job Page

JobPage displays the job’s status, group (if available), and the stages per state: active, pending, completed, skipped, and failed.

A job can be in a running, succeeded, failed or unknown state.
spark webui jobs details for job active pending stages.png
Figure 10. Details for Job Page with Active and Pending Stages
spark webui jobs details for job four stages.png
Figure 11. Details for Job Page with Four Stages

results matching ""

    No results matching ""