Vol.35, No.6, 2020, pp.523-532, doi:10.32604/csse.2020.35.523
OPEN ACCESS
ARTICLE
A Data-Aware Remote Procedure Call Method for Big Data Systems
  • Jin Wang1,2, Yaqiong Yang1, Jingyu Zhang1,3,*, Xiaofeng Yu4, Osama Alfarraj5, Amr Tolba5,6
1 School of Computer & Communication Engineering, Changsha University of Science & Technology, Changsha, 410000, China
2 School of Information Science and Engineering, Fujian University of Technology, Fujian, 350000, China
3 School of Systems Engineering, National University of Defense Technology, Changsha, 410000, China
4 School of Business, Nanjing University, Nanjing, 210093, China
5 Computer Science Department, Community College, King Saud University, Riyadh, 11437, Saudi Arabia
6 Mathematics and Computer Science Department, Faculty of Science, Menoufia University, Shebin-El-kom, 32511, Egypt
* Corresponding Author: Jingyu Zhang. Email:
Abstract
In recent years, big data has been one of the hottest development directions in the information field. With the development of artificial intelligence technology, mobile smart terminals and high-bandwidth wireless Internet, various types of data are increasing exponentially. Huge amounts of data contain a lot of potential value, therefore how to effectively store and process data efficiently becomes very important. Hadoop Distributed File System (HDFS) has emerged as a typical representative of dataintensive distributed big data file systems, and it has features such as high fault tolerance, high throughput, and can be deployed on low-cost hardwares. HDFS nodes communicate with each other to make the big data systems work properly, using the Remote Procedure Call (RPC) mechanism. However, the RPC in HDFS is still not good enough to work better in terms of network throughput and abnormal response. This paper presents an optimization method to improve the performance of HDFS. The proposed method dynamically adjusts the RPC configurations between NameNode and DataNodes by sensing the data characters that stored in DataNodes. This method can effectively reduce the NameNode processing pressure, and improve the network throughput generated by the information transmission between NameNode and DataNodes. It can also reduce the abnormal response time of the whole system. Finally, the extensive experiments show the effectiveness and efficiency of our proposed method.
Keywords
Distributed file systems; HDFS; information interaction; adaptive RPC mechanism
Cite This Article
. , "A data-aware remote procedure call method for big data systems," Computer Systems Science and Engineering, vol. 35, no.6, pp. 523–532, 2020.
This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.