Open Access

ARTICLE

An Improved Knowledge Distillation Algorithm and Its Application to Object Detection

Min Yao^1,*, Guofeng Liu², Yaozu Zhang³, Guangjie Hu¹

1 School of Information Engineering, Shanghai Maritime University, Shanghai, 201306, China
2 Baidu, Beijing, 100000, China
3 Shanghai Freesense Technology Co., Ltd., Shanghai, 200000, China

* Corresponding Author: Min Yao. Email:

Computers, Materials & Continua 2025, 83(2), 2189-2205. https://doi.org/10.32604/cmc.2025.060609

Received 06 November 2024; Accepted 08 February 2025; Issue published 16 April 2025

Abstract

Knowledge distillation (KD) is an emerging model compression technique for learning compact object detector models. Previous KD often focused solely on distilling from the logits layer or the feature intermediate layers, which may limit the comprehensive learning of the student network. Additionally, the imbalance between the foreground and background also affects the performance of the model. To address these issues, this paper employs feature-based distillation to enhance the detection performance of the bounding box localization part, and logit-based distillation to improve the detection performance of the category prediction part. Specifically, for the intermediate layer feature distillation, we introduce feature resampling to reduce the risk of the student model merely imitating the teacher model. At the same time, we incorporate a Spatial Attention Mechanism (SAM) to highlight the foreground features learned by the student model. In terms of output layer feature distillation, we divide the traditional distillation targets into target-class objects and non-target-class objects, aiming to improve overall distillation performance. Furthermore, we introduce a one-to-many matching distillation strategy based on Feature Alignment Module (FAM), which further enhances the student model’s feature representation ability, making its feature distribution closer to that of the teacher model, and thus demonstrating superior localization and classification capabilities in object detection tasks. Experimental results demonstrate that our proposed methodology outperforms conventional distillation techniques in terms of object detecting performance.

---

Keywords

---