Open Access

ARTICLE

An AI/ML Framework-Driven Approach for Malicious Traffic Detection in Open RAN

Suhyeon Lee¹, Hwankuk Kim^2,*

1 Department of Cyber Security, Kookmin University, Seoul, 02707, Republic of Korea
2 Department of Information Security Cryptography Mathematics, Kookmin University, Seoul, 02707, Republic of Korea

* Corresponding Author: Hwankuk Kim. Email:

(This article belongs to the Special Issue: Cutting-Edge Security and Privacy Solutions for Next-Generation Intelligent Mobile Internet Technologies and Applications)

Computer Modeling in Engineering & Sciences 2025, 145(2), 2657-2682. https://doi.org/10.32604/cmes.2025.070627

Received 20 July 2025; Accepted 20 October 2025; Issue published 26 November 2025

Abstract

The open nature and heterogeneous architecture of Open Radio Access Network (Open RAN) undermine the consistency of security policies and broaden the attack surface, thereby increasing the risk of security vulnerabilities. The dynamic nature of network performance and traffic patterns in Open RAN necessitates advanced detection models that can overcome the constraints of traditional techniques and adapt to evolving behaviors. This study presents a methodology for effectively detecting malicious traffic in Open RAN by utilizing an Artificial-Intelligence/Machine-Learning (AI/ML) Framework. A hybrid Transformer–Convolutional-Neural-Network (Transformer-CNN) ensemble model is employed for anomaly detection. The proposed model generates final predictions through a soft-voting technique based on the predictive outputs of the two models with distinct features. This approach improves accuracy by up to 1.06% and F1 score by 1.48% compared with a hard-voting technique to determine the final prediction. Furthermore, the proposed model achieves an average accuracy of approximately 98.3% depending on the time step, exhibiting a 1.43% increase in accuracy over single-model approaches. Unlike single-model approaches, which are prone to overfitting, the ensemble model resolves the overfitting problem by reducing the deviation in validation loss.

---

Keywords

---