Vol.30, No.2, 2021, pp.481-494, doi:10.32604/iasc.2021.019201
A Resource-constrained Edge IoT Device Data-deduplication Method with Dynamic Asymmetric Maximum
  • Ye Yang1, Xiaofang Li2, Dongjie Zhu3,*, Hao Hu3, Haiwen Du4, Yundong Sun4, Weiguo Tian3, Yansong Wang3, Ning Cao1, Gregory M.P. O’Hare5
1 School of Artificial Intelligence, Wuxi Vocational College of Science and Technology, Wuxi, 214028, China
2 Department of Mathematics, Harbin Institute of Technology, Weihai, 264209, China
3 School of Computer Science and Technology, Harbin Institute of Technology, Weihai, 264200, China
4 School of Astronautics, Harbin Institute of Technology, Harbin, 150001, China
5 School of Computer Science, University College Dublin, Dublin, Dublin4, Ireland
* Corresponding Author: Dongjie Zhu. Email:
Received 06 April 2021; Accepted 13 May 2021; Issue published 11 August 2021
Smart vehicles use sophisticated sensors to capture real-time data. Due to the weak communication capabilities of wireless sensors, these data need to upload to the cloud for processing. Sensor clouds can resolve these drawbacks. However, there is a large amount of redundant data in the sensor cloud, occupying a large amount of storage space and network bandwidth. Deduplication can yield cost savings by storing one data copy. Chunking is essential because it can determine the performance of deduplication. Content-Defined Chunking (CDC) can effectively solve the problem of chunk boundaries shifted, but it occupies a lot of computing resources and has become a bottleneck in deduplication technology. This paper proposes a Dynamic Asymmetric Maximum algorithm (DAM), which uses the maximum value as the chunk boundaries and reducing the impact of the low-entropy string. It also uses the perfect hash algorithm to optimize the chunk search. Experiments show that our solution can effectively detect low-entropy strings in redundant data, save storage resources, and improve sensor clouds system throughput.
Internet of Things; Sensor clouds; Intelligent transportation; Data deduplication
Cite This Article
Yang, Y., Li, X., Zhu, D., Hu, H., Du, H. et al. (2021). A Resource-constrained Edge IoT Device Data-deduplication Method with Dynamic Asymmetric Maximum. Intelligent Automation & Soft Computing, 30(2), 481–494.
This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.