Open Access
ARTICLE
An Optimized Resource Scheduling Strategy for Hadoop Speculative Execution Based on Non-cooperative Game Schemes
Yinghang Jiang1, Qi Liu2,3,*, Williams Dannah1, Dandan Jin2, Xiaodong Liu3, Mingxu Sun4,*
1 Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology (CICAEET), Nanjing University of Information Science & Technology, Nanjing, 210044, China.
2 School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing, 210044, China.
3 School of Computing, Edinburgh Napier University, Edinburgh EH10 5DT, UK.
4 School of Electrical Engineering, University of Jinan, China, and Centre for Health Sciences Research, University of Salford, Salford, Greater Manchester, M5 4WT, UK.
* Corresponding Authors: Qi Liu. Email: ;
Mingxu Sun. Email: .
Computers, Materials & Continua 2020, 62(2), 713-729. https://doi.org/10.32604/cmc.2020.04604
Abstract
Hadoop is a well-known parallel computing system for distributed computing
and large-scale data processes. “Straggling” tasks, however, have a serious impact on task
allocation and scheduling in a Hadoop system. Speculative Execution (SE) is an efficient
method of processing “Straggling” Tasks by monitoring real-time running status of tasks
and then selectively backing up “Stragglers” in another node to increase the chance to
complete the entire mission early. Present speculative execution strategies meet challenges
on misjudgement of “Straggling” tasks and improper selection of backup nodes, which
leads to inefficient implementation of speculative executive processes. This paper has
proposed an Optimized Resource Scheduling strategy for Speculative Execution (ORSE)
by introducing non-cooperative game schemes. The ORSE transforms the resource
scheduling of backup tasks into a multi-party non-cooperative game problem, where the
tasks are regarded as game participants, whilst total task execution time of the entire cluster
as the utility function. In that case, the most benefit strategy can be implemented in each
computing node when the game reaches a Nash equilibrium point, i.e., the final resource
scheduling scheme to be obtained. The strategy has been implemented in Hadoop-2.x.
Experimental results depict that the ORSE can maintain the efficiency of speculative
executive processes and improve fault-tolerant and computation performance under the
circumstances of Normal Load, Busy Load and Busy Load with Skewed Data.
Keywords
Cite This Article
Y. Jiang, Q. Liu, W. Dannah, D. Jin, X. Liu
et al., "An optimized resource scheduling strategy for hadoop speculative execution based on non-cooperative game schemes,"
Computers, Materials & Continua, vol. 62, no.2, pp. 713–729, 2020. https://doi.org/10.32604/cmc.2020.04604
Citations