TY - EJOU AU - Alhashmi, Asma A. AU - Darem, Abdulbasit A. AU - Alanazi, Sultan M. AU - Alashjaee, Abdullah M. AU - Aldughayfiq, Bader AU - Ghaleb, Fuad A. AU - Ebad, Shouki A. AU - Alanazi, Majed A. TI - Hybrid Malware Variant Detection Model with Extreme Gradient Boosting and Artificial Neural Network Classifiers T2 - Computers, Materials \& Continua PY - 2023 VL - 76 IS - 3 SN - 1546-2226 AB - In an era marked by escalating cybersecurity threats, our study addresses the challenge of malware variant detection, a significant concern for a multitude of sectors including petroleum and mining organizations. This paper presents an innovative Application Programmable Interface (API)-based hybrid model designed to enhance the detection performance of malware variants. This model integrates eXtreme Gradient Boosting (XGBoost) and an Artificial Neural Network (ANN) classifier, offering a potent response to the sophisticated evasion and obfuscation techniques frequently deployed by malware authors. The model’s design capitalizes on the benefits of both static and dynamic analysis to extract API-based features, providing a holistic and comprehensive view of malware behavior. From these features, we construct two XGBoost predictors, each of which contributes a valuable perspective on the malicious activities under scrutiny. The outputs of these predictors, interpreted as malicious scores, are then fed into an ANN-based classifier, which processes this data to derive a final decision. The strength of the proposed model lies in its capacity to leverage behavioral and signature-based features, and most importantly, in its ability to extract and analyze the hidden relations between these two types of features. The efficacy of our proposed API-based hybrid model is evident in its performance metrics. It outperformed other models in our tests, achieving an impressive accuracy of 95% and an F-measure of 93%. This significantly improved the detection performance of malware variants, underscoring the value and potential of our approach in the challenging field of cybersecurity. KW - API-based hybrid malware; detection model; static and dynamic analysis; malware detection DO - 10.32604/cmc.2023.041038