Abstract-Many companies start using Hadoop for advanced data analytics over large datasets. While a traditional Hadoop cluster deployment assumes a homogeneous cluster, many enterprise clusters are grown incrementally over time, and might have a variety of different servers in the cluster. The nodes' heterogeneity represents an additional challenge for efficient cluster and job management. Due to resource heterogeneity, it is often unclear which resources introduce inefficiency and bottlenecks, and how such a Hadoop cluster should be configured and optimized. In this work 1 , we explore the efficiency and performance accuracy of the bounds-based performance model for predicting the MapReduce job completion times in heterogeneous Hadoop clusters. We validate the accuracy of the proposed performance model using a diverse set of 13 realistic applications and two different heterogeneous clusters. Since one of the Hadoop clusters is formed by different capacity VM instances in Amazon EC2 environment, we additionally explore and discuss factors that impact the MapReduce job performance in the Cloud. Keywords-MapReduce, heterogeneous clusters, performance modeling, efficiency. In recent years, there has been an emergence of several data-parallel middleware platforms primarily targeted towards large-scale data processing in the cloud environment, for example, MapReduce [3] and its popular open-source Hadoop implementation. These middleware platforms provide simple yet powerful and extremely versatile programming models, and can be applied to a wide spectrum of domains where large-scale data analytics is performed. To better manage cloud resources, there has been a series of efforts aimed at developing performance models [4] These performance models are used for predicting the job completion times resources. They are also used as a basis for the resource provisioning problem: and estimating the amount of resources required to complete applications with given deadlines. One major shortcoming of the existing models is that they do not consider the resource heterogeneity which causes inaccurate prediction on heterogeneous clusters. We argue that cluster deployments are grown incrementally over time. It is not unusual for companies to start off with an initial cluster, and then gradually add more compute and I/O resources as the number of users increases. More often than not, it is economical to add newer more powerful servers rather than discard the old hardware. So, the problem is to design and offer the efficient performance models that work well in heterogeneous environments. Performance models that rely on extensive fine-granularity profiling techniques [4], [5] or complex analysis using ma-
Zhang et al. REF attempted to base distributed computing jobs on heterogeneous machines and to predict job completion time based on boundary-based performance modeling.
Performance Modeling of MapReduce Jobs in Heterogeneous Cloud Environments
2013 IEEE Sixth International Conference on Cloud Computing
2013 IEEE Sixth International Conference on Cloud Computing
Computer Science