Table of Content

Open Access iconOpen Access


A Spark Scheduling Strategy for Heterogeneous Cluster

Xuewen Zhang1, Zhonghao Li1, Gongshen Liu1,*, Jiajun Xu1, Tiankai Xie2, Jan Pan Nees1

School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, P.R.C.
Eberly College of Science, Pennsylvania State University, Old Main, State College, PA 16801, U.S.A.

* Corresponding Author: Gongshen Liu. Email: email.

Computers, Materials & Continua 2018, 55(3), 405-417.


As a main distributed computing system, Spark has been used to solve problems with more and more complex tasks. However, the native scheduling strategy of Spark assumes it works on a homogenized cluster, which is not so effective when it comes to heterogeneous cluster. The aim of this study is looking for a more effective strategy to schedule tasks and adding it to the source code of Spark. After investigating Spark scheduling principles and mechanisms, we developed a stratifying algorithm and a node scheduling algorithm is proposed in this paper to optimize the native scheduling strategy of Spark. In this new strategy, the static level of nodes is calculated, the dynamic factors such as the length of running tasks, and CPU usage of work nodes are considered comprehensively. And through a series of comparative experiments in alienation cluster, the new strategy costs less running time and lower CPU usage rate than the original Spark strategy, which verifies that the new schedule strategy is more effective one.


Cite This Article

APA Style
Zhang, X., Li, Z., Liu, G., Xu, J., Xie, T. et al. (2018). A spark scheduling strategy for heterogeneous cluster. Computers, Materials & Continua, 55(3), 405-417.
Vancouver Style
Zhang X, Li Z, Liu G, Xu J, Xie T, Nees JP. A spark scheduling strategy for heterogeneous cluster. Comput Mater Contin. 2018;55(3):405-417
IEEE Style
X. Zhang, Z. Li, G. Liu, J. Xu, T. Xie, and J.P. Nees "A Spark Scheduling Strategy for Heterogeneous Cluster," Comput. Mater. Contin., vol. 55, no. 3, pp. 405-417. 2018.

cc This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 2181


  • 1380


  • 0


Related articles

Share Link