Suzhen Wang1, Shanshan Geng1, Zhanfeng Zhang1, Anshan Ye2, Keming Chen2, Zhaosheng Xu2, Huimin Luo2, Gangshan Wu3,*, Lina Xu4, Ning Cao5
CMC-Computers, Materials & Continua, Vol.61, No.2, pp. 739-757, 2019, DOI:10.32604/cmc.2019.06097
Abstract Spark is a distributed data processing framework based on memory. Memory allocation is a focus question of Spark research. A good memory allocation scheme can effectively improve the efficiency of task execution and memory resource utilization of the Spark. Aiming at the memory allocation problem in the Spark2.x version, this paper optimizes the memory allocation strategy by analyzing the Spark memory model, the existing cache replacement algorithms and the memory allocation methods, which is on the basis of minimizing the storage area and allocating the execution area according to the demand. It mainly including two parts: cache replacement optimization and… More >
Login
Register
