TY  - EJOU
AU  - Yao, Min 
AU  - Liu, Guofeng 
AU  - Zhang, Yaozu 
AU  - Hu, Guangjie 

TI  - An Improved Knowledge Distillation Algorithm and Its Application to Object Detection
T2  - Computers, Materials \& Continua

PY  - 2025
VL  - 83
IS  - 2
SN  - 1546-2226

AB  - Knowledge distillation (KD) is an emerging model compression technique for learning compact object detector models. Previous KD often focused solely on distilling from the logits layer or the feature intermediate layers, which may limit the comprehensive learning of the student network. Additionally, the imbalance between the foreground and background also affects the performance of the model. To address these issues, this paper employs feature-based distillation to enhance the detection performance of the bounding box localization part, and logit-based distillation to improve the detection performance of the category prediction part. Specifically, for the intermediate layer feature distillation, we introduce feature resampling to reduce the risk of the student model merely imitating the teacher model. At the same time, we incorporate a Spatial Attention Mechanism (SAM) to highlight the foreground features learned by the student model. In terms of output layer feature distillation, we divide the traditional distillation targets into target-class objects and non-target-class objects, aiming to improve overall distillation performance. Furthermore, we introduce a one-to-many matching distillation strategy based on Feature Alignment Module (FAM), which further enhances the student model’s feature representation ability, making its feature distribution closer to that of the teacher model, and thus demonstrating superior localization and classification capabilities in object detection tasks. Experimental results demonstrate that our proposed methodology outperforms conventional distillation techniques in terms of object detecting performance.
KW  - Deep learning; model compression; knowledge distillation; object detection

DO  - 10.32604/cmc.2025.060609