This article discusses problems and decisions related to scheduling of stream processing applications in heterogeneous clusters. An overview of the current state of the art of the stream processing on heterogeneous clusters with a focus on resource allocation and scheduling is presented first. Then, common scheduling approaches of various stream processing frameworks are discussed and their limited applicability in the heterogeneous environment is demonstrated on a simple stream application. Finally, the article presents a novel heterogeneity-aware scheduler for the stream processing frameworks based on design-time knowledge as well as benchmarking techniques. It is shown that the scheduler overcomes alternatives in resource-aware deployment over cluster nodes and thus it leads to a better utilisation of the clusters. Keywords: scheduling; resource-awareness; benchmarking; stream processing; Apache Storm; heterogeneous clusters. Reference to this paper should be made as follows: Rychlý, M., Škoda, P. and Smrž, P. Petr Škoda is PhD student and junior researcher at BUT FIT. He is responsible for the design and development of a large-scale applications based on the Hadoop and Storm frameworks. His research further tackles resource allocation and scheduling, distributed computing, and architecture of software systems. He authored three papers in scientific conference proceedings. Pavel Smrž is an Associate Professor at BUT FIT. He leads the Knowledge Technology Research Group that has participated in many European as well as national research and development projects. His research interests include semantic technologies, large-scale distributed systems, deep learning, hardware-accelerated computing, natural language processing and human machine interaction. He has authored more than 60 papers in scientific journals and in conference proceedings. This paper is a revised and expanded version of a paper entitled 'Scheduling decisions in stream processing on heterogeneous clusters' presented at
Rychl et al. REF presented a heterogeneity-aware scheduler for the stream processing frameworks.
Heterogeneity-aware scheduler for stream processing frameworks
Int. J. Big Data Intell.
Int. J. Big Data Intell.
Computer Science