Open Access

REVIEW

A Survey of AI-Based Encrypted Traffic Detection: Multi-Level Taxonomy and Structural Analysis of Intent–Behavior–Model Coupling

Yeog Kim, Changhoon Lee, Kiwook Sohn^*

Cryptographic and Information Security Laboratory, Seoul National University of Science and Technology, Seoul, Republic of Korea

* Corresponding Author: Kiwook Sohn. Email:

(This article belongs to the Special Issue: The Evolution of Cybersecurity and AI: Surveys and Tutorials)

Computer Modeling in Engineering & Sciences 2026, 148(1), 5 https://doi.org/10.32604/cmes.2026.083669

Received 08 April 2026; Accepted 29 May 2026; Issue published 27 July 2026

Abstract

With the widespread adoption of encryption protocols, payload-based traffic analysis has become increasingly infeasible, posing significant challenges for intrusion detection systems (IDS). Consequently, AI-based approaches for encrypted traffic analysis have gained substantial attention. However, existing studies are often evaluated using inconsistent criteria, including heterogeneous attack labels, behavioral representations, and model architectures, making systematic comparison difficult. To address this limitation, this paper proposes a three-level analytical taxonomy for encrypted traffic analysis, structured around attack objectives (Level 1), observable network behaviors (Level 2), and detection models (Level 3). The proposed framework provides a structured perspective for analyzing how detection objectives, behavioral abstractions, and model design interact under encryption constraints. Based on a systematic analysis of 53 representative studies, this survey examines the relationship between attack objectives, behavior patterns, datasets, evaluation metrics, and AI-based detection models. The analysis indicates that behavioral patterns play an important role in connecting attack objectives with detection models, while also revealing imbalances in the coverage of attack objectives across existing studies. In addition, the survey highlights how dataset selection and evaluation criteria influence the interpretation of model performance in encrypted traffic analysis. Overall, the proposed taxonomy provides a behavior-centric analytical framework for organizing existing encrypted traffic analysis studies and offers insights for future IDS research in encrypted network environments.

---

Keywords

---