Deep Reinforcement Learning-Based Intrusion Detection in IoT Networks: A Systematic Mapping and Literature Review

Maryam Omar Abdullah Sawad¹, Said Jadid Abdulkadir^1,2,*, Hitham Seddig Alhussian^1,2, Majdy Mohamed Eltayeb Eltahir³
1 Department of Computing, Universiti Teknologi PETRONAS, Seri Iskandar, Perak, Malaysia
2 Center for Research in Data Science (CeRDaS), Universiti Teknologi PETRONAS, Seri Iskandar, Perak, Malaysia
3 Unit of Specialization of Technology and Engineering, Applied College of Muhayil Asir, King Khalid University, Muhayil Asir, Saudi Arabia
* Corresponding Author: Said Jadid Abdulkadir. Email: email

Computer Modeling in Engineering & Sciences https://doi.org/10.32604/cmes.2026.085472

Received 11 May 2026; Accepted 29 June 2026; Published online 31 July 2026

Abstract

The increasing complexity and heterogeneity of cyberattacks targeting Internet of Things (IoT) environments, driven by the diversity of interconnected nodes and communication channels, necessitate the development of more advanced and intelligent cyber defence techniques. However, the most effective methods are Machine Learning (ML)-based and Deep Learning (DL)-based intrusion detection systems (IDS), which perform well but still face significant limitations and challenges. To address these issues, Deep Reinforcement Learning (DRL) has been proposed in recent years to automatically resolve the issues by detecting attacks in IoT environments. Therefore, this Systematic Literature Review (SLR) presents an up-to-date review by analyzing the existing studies on DRL-based IDS models that detect intrusions in IoT networks. To achieve this goal, this review focuses on and scrutinizes scientific journals and articles extracted from 2020 to 2026 across multiple databases, identifying 267 articles. A systematic mapping procedure was then carried out using Rayyan and Mendeley to screen the articles based on nine well-defined inclusion and exclusion criteria covering publication year, language, publication type, full-text availability, explicit use of DRL techniques, relevance to IoT attack detection, minimum page count, duplication, and open access, which collectively reduced the pool to 26 high-quality studies. The majority of excluded articles 241 in total were removed primarily because they did not explicitly employ DRL techniques in an IoT intrusion detection context, were duplicate records, or did not meet the minimum quality thresholds defined in the eligibility assessment. The review reveals that the most used algorithm for DRL-based IDS is Deep Q-Network (DQN), appearing in 8 studies (30.8%). Three studies utilized feature selection methods, including LightGBM and Mutual Information Feature Selection (MIFS), and the most frequently targeted attacks are DoS, DDoS, Backdoors, Mirai, Reconnaissance, Scan, and Torii. Finally, this research highlights the open issues and challenges for future research in DRL-based IDS models, to enhance IoT network security.