TY - EJOU AU - Gupta, Brij B. AU - Gaurav, Akshat AU - Alhalabi, Wadee AU - Arya, Varsha AU - Bansal, Shavi AU - Hsu, Ching-Hsien TI - AI-Driven Malware Detection with VGG Feature Extraction and Artificial Rabbits Optimized Random Forest Model T2 - Computers, Materials \& Continua PY - 2025 VL - 84 IS - 3 SN - 1546-2226 AB - Detecting cyber attacks in networks connected to the Internet of Things (IoT) is of utmost importance because of the growing vulnerabilities in the smart environment. Conventional models, such as Naive Bayes and support vector machine (SVM), as well as ensemble methods, such as Gradient Boosting and eXtreme gradient boosting (XGBoost), are often plagued by high computational costs, which makes it challenging for them to perform real-time detection. In this regard, we suggested an attack detection approach that integrates Visual Geometry Group 16 (VGG16), Artificial Rabbits Optimizer (ARO), and Random Forest Model to increase detection accuracy and operational efficiency in Internet of Things (IoT) networks. In the suggested model, the extraction of features from malware pictures was accomplished with the help of VGG16. The prediction process is carried out by the random forest model using the extracted features from the VGG16. Additionally, ARO is used to improve the hyper-parameters of the random forest model of the random forest. With an accuracy of 96.36%, the suggested model outperforms the standard models in terms of accuracy, F1-score, precision, and recall. The comparative research highlights our strategy’s success, which improves performance while maintaining a lower computational cost. This method is ideal for real-time applications, but it is effective. KW - Malware detection; VGG feature extraction; artificial rabbits; optimization; random forest model DO - 10.32604/cmc.2025.064053