is a revolutionary network architecture that separates out network control functions from the underlying equipment and is an increasing trend to help enterprises build more manageable data centers where big data processing emerges as an important part of applications. To concurrently process large-scale data, MapReduce with an opensource implementation named Hadoop is proposed. In practical Hadoop systems, one kind of issue that vitally impacts the overall performance is known as the NP-complete minimum make span 1 problem. One main solution is to assign tasks on data local nodes to avoid link occupation since network bandwidth is a scarce resource. Many methodologies for enhancing data locality are proposed such as the Hadoop default scheduler (HDS) and stateof-the-art scheduler balance-reduce scheduler (BAR). However, all of them either ignore allocating tasks in a global view or disregard available bandwidth as the basis for scheduling. In this paper, we propose a heuristic bandwidth-aware task scheduler bandwidthaware scheduling with SDN in Hadoop (BASS) to combine Hadoop with SDN. It is not only able to guarantee data locality in a global view but also can efficiently assign tasks in an optimized way. Both examples and experiments demonstrate that BASS has the best performance in terms of job completion time. To our knowledge, BASS is the first to exploit talent of SDN for job scheduling of big data processing and we believe that it points out a new trend for large-scale data processing.
Qin et al. in a recent study REF proposed a bandwidth-aware scheduling with SDN (BASS) mechanism for Hadoop jobs that can improve the performance in terms of job turnaround time.
Bandwidth-Aware Scheduling With SDN in Hadoop: A New Trend for Big Data
IEEE Systems Journal
IEEE Systems Journal
Computer Science