Volume 15 - Issue 3
Designing Effective Schemes for Static Slot Configuration in Hadoop
Abstract
The MapReduce framework and the non-proprietary distributed processing framework Hadoop have developed as an important platform for analysing huge data sets in the last few years. One of the chief tasks is reducing the makespan of map and reduce jobs in Hadoop. The existing Hadoop system allows only static slot configuration in which the number of map slots and reduce slots of each of the node in a cluster are fixed throughout its lifetime. This assignment does not depend upon the characteristics of incoming jobs and their completion time. This technique uses FIFO and fair share scheduling algorithms. Such a rigid configuration may lead to little utilization of system resources and also longer makespan. Motivated by this, we propose easy yet successful reconfigurable schemes which uses a tunable knob for slot assignment ratio of map slots and reduce slots for decreasing the makespan of a given dataset. Our strategies dynamically assign map tasks and reduce tasks by using the workload information of freshly finished jobs. The experimental results illustrate the productiveness and activeness of our schemes under both homogeneous clusters having elementary workloads and heterogeneous clusters having more convoluted workloads.
Paper Details
PaperID: 191052
Author's Name: D. Jagadeesh Sai, Parinitha Srinivas, S.S. Shreedeebika, Surbha Kappor and Yashi Gupta
Volume: Volume 15
Issues: Issue 3
Keywords: Hadoop, MapReduce, Makespan, Job Tracker, Task Tracker.
Year: 2019
Month: May
Pages: 98-104