Designing Effective Schemes for Static Slot Configuration in Hadoop
The MapReduce framework and the non-proprietary distributed processing framework Hadoop have developed as an important platform for analysing huge data sets in the last few years. One of the chief tasks is reducing the makespan of map and reduce jobs in Hadoop. The existing Hadoop system allows only static slot configuration in which the number of map slots and reduce slots of each of the node in a cluster are fixed throughout its lifetime. This assignment does not depend upon the characteristics of incoming jobs and their completion time. This technique uses FIFO and fair share scheduling algorithms. Such a rigid configuration may lead to little utilization of system resources and also longer makespan. Motivated by this, we propose easy yet successful reconfigurable schemes which uses a tunable knob for slot assignment ratio of map slots and reduce slots for decreasing the makespan of a given dataset. Our strategies dynamically assign map tasks and reduce tasks by using the workload information of freshly finished jobs. The experimental results illustrate the productiveness and activeness of our schemes under both homogeneous clusters having elementary workloads and heterogeneous clusters having more convoluted workloads.
Author's Name: D. Jagadeesh Sai, Parinitha Srinivas, S.S. Shreedeebika, Surbha Kappor and Yashi Gupta