Job aware scheduling in Hadoop for heterogeneous cluster

• Published in: 2015 IEEE International Advance Computing Conference (IACC) pp. 778 - 783
• Main Authors: Pati, Supriya, Mehta, Mayuri A.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-06-2015
• Subjects: Clustering algorithms; Hadoop; Heart beat; job aware; MapReduce; Resource management; Schedules; Scheduling; Scheduling algorithms
• DOI: 10.1109/IADCC.2015.7154813

Hadoop cluster is specifically designed to store and analyze a large amount of data in distributed environment. With ever increasing use of Hadoop clusters, a scheduling algorithm is required for optimal utilisation of cluster resources. The existing scheduling algorithms are limited to one or more of the following crucial problems such as limited utilization of computing resources, limited applicability towards heterogeneous cluster, random scheduling of non-local map tasks, and negligence of small jobs in scheduling. In this paper, we propose a novel job aware scheduling algorithm that overcomes the above limitations. In addition, we analyze the performance of the proposed algorithm using MapReduce WordCount benchmark. The experimental results show that the proposed algorithm increases the resource utilization and reduces the average waiting time compared to existing Matchmaking scheduling algorithm.