Dynamic Load Balancing and Task Scheduling Optimization in Hadoop Clusters

Abstract

Hadoop is a widely utilized distributed file system and processing framework for handling large-scale data. Nonetheless, the inherent load balancing and task scheduling mechanisms in Hadoop exhibit inefficiencies that may result in performance bottlenecks. In this paper, we propose a novel dynamic load-balancing algorithm designed specifically for Hadoop clusters. Our algorithm continuously monitors the performance indicators of nodes and dynamically adjusts task-node allocations to ensure equitable load distribution within the cluster. Furthermore, we consider the execution states of tasks to optimize resource allocation effectively. The primary contribution of this study resides in the analysis and resolution of load balancing and scheduling issues within Hadoop. In addition, our proposed dynamic scheduling algorithm also accounts for task execution states, thereby facilitating optimized resource allocation. We validate our algorithm across various workloads, demonstrating that it surpasses existing methods in job completion time, scalability, and resource utilization. The findings indicate that the proposed algorithm efficiently balances cluster loads, expedites task completion, and reduces both costs and resource consumption.