Open Access


A Resource-constrained Edge IoT Device Data-deduplication Method with Dynamic Asymmetric Maximum

Ye Yang1, Xiaofang Li2, Dongjie Zhu3,*, Hao Hu3, Haiwen Du4, Yundong Sun4, Weiguo Tian3, Yansong Wang3, Ning Cao1, Gregory M.P. O’Hare5
1 School of Artificial Intelligence, Wuxi Vocational College of Science and Technology, Wuxi, 214028, China
2 Department of Mathematics, Harbin Institute of Technology, Weihai, 264209, China
3 School of Computer Science and Technology, Harbin Institute of Technology, Weihai, 264200, China
4 School of Astronautics, Harbin Institute of Technology, Harbin, 150001, China
5 School of Computer Science, University College Dublin, Dublin, Dublin4, Ireland
* Corresponding Author: Dongjie Zhu. Email:

Intelligent Automation & Soft Computing 2021, 30(2), 481-494.

Received 06 April 2021; Accepted 13 May 2021; Issue published 11 August 2021


Smart vehicles use sophisticated sensors to capture real-time data. Due to the weak communication capabilities of wireless sensors, these data need to upload to the cloud for processing. Sensor clouds can resolve these drawbacks. However, there is a large amount of redundant data in the sensor cloud, occupying a large amount of storage space and network bandwidth. Deduplication can yield cost savings by storing one data copy. Chunking is essential because it can determine the performance of deduplication. Content-Defined Chunking (CDC) can effectively solve the problem of chunk boundaries shifted, but it occupies a lot of computing resources and has become a bottleneck in deduplication technology. This paper proposes a Dynamic Asymmetric Maximum algorithm (DAM), which uses the maximum value as the chunk boundaries and reducing the impact of the low-entropy string. It also uses the perfect hash algorithm to optimize the chunk search. Experiments show that our solution can effectively detect low-entropy strings in redundant data, save storage resources, and improve sensor clouds system throughput.


Internet of Things; Sensor clouds; Intelligent transportation; Data deduplication

Cite This Article

Y. Yang, X. Li, D. Zhu, H. Hu, H. Du et al., "A resource-constrained edge iot device data-deduplication method with dynamic asymmetric maximum," Intelligent Automation & Soft Computing, vol. 30, no.2, pp. 481–494, 2021.

This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 874


  • 492


  • 0


Share Link

WeChat scan