Open Access iconOpen Access



Run-Time Dynamic Resource Adjustment for Mitigating Skew in MapReduce

Zhihong Liu1, Shuo Zhang2,*, Yaping Liu2, Xiangke Wang1, Dong Yin1

1 College of Intelligence and Technology, National University of Defense Technology, Changsha, 410073, China
2 Cyberspace Institute of Advanced Technology, Guangzhou University, Guangzhou, 510006, China

* Corresponding Author: Shuo Zhang. Email: email

(This article belongs to the Special Issue: Intelligent Models for Security and Resilience in Cyber Physical Systems)

Computer Modeling in Engineering & Sciences 2021, 126(2), 771-790.


MapReduce is a widely used programming model for large-scale data processing. However, it still suffers from the skew problem, which refers to the case in which load is imbalanced among tasks. This problem can cause a small number of tasks to consume much more time than other tasks, thereby prolonging the total job completion time. Existing solutions to this problem commonly predict the loads of tasks and then rebalance the load among them. However, solutions of this kind often incur high performance overhead due to the load prediction and rebalancing. Moreover, existing solutions target the partitioning skew for reduce tasks, but cannot mitigate the computational skew for map tasks. Accordingly, in this paper, we present DynamicAdjust, a run-time dynamic resource adjustment technique for mitigating skew. Rather than rebalancing the load among tasks, DynamicAdjust monitors the run-time execution of tasks and dynamically increases resources for those tasks that require more computation. In so doing, DynamicAdjust can not only eliminate the overhead incurred by load prediction and rebalancing, but also culls both the partitioning skew and the computational skew. Experiments are conducted based on a 21-node real cluster using real-world datasets. The results show that DynamicAdjust can mitigate the negative impact of the skew and shorten the job completion time by up to 40.85%.


Cite This Article

APA Style
Liu, Z., Zhang, S., Liu, Y., Wang, X., Yin, D. (2021). Run-time dynamic resource adjustment for mitigating skew in mapreduce. Computer Modeling in Engineering & Sciences, 126(2), 771-790.
Vancouver Style
Liu Z, Zhang S, Liu Y, Wang X, Yin D. Run-time dynamic resource adjustment for mitigating skew in mapreduce. Comput Model Eng Sci. 2021;126(2):771-790
IEEE Style
Z. Liu, S. Zhang, Y. Liu, X. Wang, and D. Yin "Run-Time Dynamic Resource Adjustment for Mitigating Skew in MapReduce," Comput. Model. Eng. Sci., vol. 126, no. 2, pp. 771-790. 2021.

cc This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 2269


  • 1436


  • 0


Share Link