Privacy-Aware Anomaly Detection in Encrypted Network Traffic via Adaptive Homomorphic Encryption

Yu-Ran Jeon¹, Seung-Ha Jee¹, Su-Kyoung Kim¹, Il-Gu Lee^1,2,*
1 Department of Future Convergence Technology Engineering, Sungshin Women’s University, Seoul, Republic of Korea
2 Department of Convergence Security Engineering, Sungshin Women’s University, Seoul, Republic of Korea
* Corresponding Author: Il-Gu Lee. Email: email
(This article belongs to the Special Issue: Advanced Security and Privacy for Future Mobile Internet and Convergence Applications: A Computer Modeling Approach)

Computer Modeling in Engineering & Sciences https://doi.org/10.32604/cmes.2026.077784

Received 16 December 2025; Accepted 23 February 2026; Published online 13 March 2026

Download PDF

Abstract

As cyberattacks become increasingly sophisticated and intelligent, demand for machine-learning-based anomaly detection systems is growing. However, conventional systems generally assume a trusted server environment, where traffic data is collected and analyzed in plaintext. This assumption introduces inherent privacy risks, as privacy-sensitive information may be exposed if the server is compromised or misused. To address this limitation, privacy-preserving anomaly detection approaches have been actively studied, enabling anomaly detection to be performed directly on encrypted traffic without revealing privacy-sensitive data. While these approaches offer strong confidentiality guarantees, they suffer from significant drawbacks, including substantial computational overhead, high latency, and degraded detection accuracy. To overcome these limitations, we propose a privacy-aware anomaly detection (PAAD) model that adaptively applies homomorphic encryption based on the privacy sensitivity of incoming traffic. Instead of encrypting all data indiscriminately, PAAD dynamically determines whether traffic should be processed in plaintext or ciphertext and performs homomorphic inference only for privacy-sensitive data. This selective encryption strategy effectively balances privacy protection and system efficiency. Extensive experiments conducted under diverse network environments demonstrate that the proposed PAAD model significantly outperforms conventional anomaly detection models. In particular, PAAD improves detection accuracy by up to 73%, reduces latency by up to 8.6 times, and achieves negligible information leakage, highlighting its practicality for real-world privacy-sensitive network monitoring scenarios.