  Distributed data processing systems like MapReduce, Spark, and Flink are
popular tools for analysis of large datasets with cluster resources. Yet, users
often overprovision resources for their data processing jobs, while the
resource usage of these jobs also typically fluctuates considerably. Therefore,
multiple jobs usually get scheduled onto the same shared resources to increase
the resource utilization and throughput of clusters. However, job runtimes and
the utilization of shared resources can vary significantly depending on the
specific combinations of co-located jobs.
  This paper presents Hugo, a cluster scheduler that continuously learns how
efficiently jobs share resources, considering metrics for the resource
utilization and interference among co-located jobs. The scheduler combines
offline grouping of jobs with online reinforcement learning to provide a
scheduling mechanism that efficiently generalizes from specific monitored job
combinations yet also adapts to changes in workloads. Our evaluation of a
prototype shows that the approach can reduce the runtimes of exemplary Spark
jobs on a YARN cluster by up to 12.5%, while resource utilization is increased
and waiting times can be bounded.
