Open Access

ARTICLE

Enhancing Malware Detection Resilience: A U-Net GAN Denoising Framework for Image-Based Classification

Huiyao Dong¹, Igor Kotenko^2,*

1 Faculty of Information Technology and Security, ITMO National Research University, St. Petersburg, 197101, Russia
2 Laboratory of Computer Security Problems, St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, 199178, Russia

* Corresponding Author: Igor Kotenko. Email:

(This article belongs to the Special Issue: Securing the Future: Innovations and Challenges in Next-Generation Network Security)

Computers, Materials & Continua 2025, 82(3), 4263-4285. https://doi.org/10.32604/cmc.2025.062439

Received 18 December 2024; Accepted 26 January 2025; Issue published 06 March 2025

Abstract

The growing complexity of cyber threats requires innovative machine learning techniques, and image-based malware classification opens up new possibilities. Meanwhile, existing research has largely overlooked the impact of noise and obfuscation techniques commonly employed by malware authors to evade detection, and there is a critical gap in using noise simulation as a means of replicating real-world malware obfuscation techniques and adopting denoising framework to counteract these challenges. This study introduces an image denoising technique based on a U-Net combined with a GAN framework to address noise interference and obfuscation challenges in image-based malware analysis. The proposed methodology addresses existing classification limitations by introducing noise addition, which simulates obfuscated malware, and denoising strategies to restore robust image representations. To evaluate the approach, we used multiple CNN-based classifiers to assess noise resistance across architectures and datasets, measuring significant performance variation. Our denoising technique demonstrates remarkable performance improvements across two multi-class public datasets, MALIMG and BIG-15. For example, the MALIMG classification accuracy improved from 23.73% to 88.84% with denoising applied after Gaussian noise injection, demonstrating robustness. This approach contributes to improving malware detection by offering a robust framework for noise-resilient classification in noisy conditions.

---

Keywords

---