YARN

Yet Another Resource Negotiator

YARN Architecture

Launching an application

yarn jar application.jar DriverClass input output

List running jobs

yarn application -list

yarn top

See application logs

yarn logs -applicationId applicationId

Kill an application

yarn application -kill applicationId

Fair Scheduler

• Resources will be shared with the rest of users using the YARN fair share scheduler
• Dominant Resource Fairness: both CPU and memory considered
• Jobs should be composed of lots of short running tasks so they share resources nicely with other jobs
• Long running tasks that monopolize resources during large times can be preempted to allow other applications to run

Fair Scheduler Queues

• root.users.[username]: default queue, one per user
• interactive: Jupyter Notebooks and interactive jobs
• urgent: limited resources that can be used for urgent jobs

Web Job Browser

You can access the HUE Job Browser from the WebUI: