SHadoop: Improving MapReduce performance by optimizing job execution mechanism in Hadoop clusters
Analyzed and identified two critical limitations of MapReduce execution mechanism. Achieved first optimization by implementing new job setup/cleanup tasks. Replaced heartbeat with an instant messaging mechanism to speedup task scheduling. Conducted comprehensive benchmarks to evaluate stable performance improvements. Passed a production test and integrated our work into Intel Distributed Hadoop.
mapreduce parallel computing performance optimization distributed processing cloud computing