Open Access

ARTICLE

A Hybrid Approach to Software Testing Efficiency: Stacked Ensembles and Deep Q-Learning for Test Case Prioritization and Ranking

Anis Zarrad¹, Thomas Armstrong², Jaber Jemai^3,*

1 School of Computer Science, University of Birmingham, Dubai, 73000, United Arab Emirates
2 School of Computer Science, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
3 Computing and Information Systems Division, Higher Colleges Technology, Abu Dhabi, 20015, United Arab Emirates

* Corresponding Author: Jaber Jemai. Email:

(This article belongs to the Special Issue: AI-Powered Software Engineering)

Computers, Materials & Continua 2026, 86(3), 74 https://doi.org/10.32604/cmc.2025.072768

Received 03 September 2025; Accepted 21 October 2025; Issue published 12 January 2026

Abstract

Test case prioritization and ranking play a crucial role in software testing by improving fault detection efficiency and ensuring software reliability. While prioritization selects the most relevant test cases for optimal coverage, ranking further refines their execution order to detect critical faults earlier. This study investigates machine learning techniques to enhance both prioritization and ranking, contributing to more effective and efficient testing processes. We first employ advanced feature engineering alongside ensemble models, including Gradient Boosted, Support Vector Machines, Random Forests, and Naive Bayes classifiers to optimize test case prioritization, achieving an accuracy score of 0.98847 and significantly improving the Average Percentage of Fault Detection (APFD). Subsequently, we introduce a deep Q-learning framework combined with a Genetic Algorithm (GA) to refine test case ranking within priority levels. This approach achieves a rank accuracy of 0.9172, demonstrating robust performance despite the increasing computational demands of specialized variation operators. Our findings highlight the effectiveness of stacked ensemble learning and reinforcement learning in optimizing test case prioritization and ranking. This integrated approach improves testing efficiency, reduces late-stage defects, and improves overall software stability. The study provides valuable information for AI-driven testing frameworks, paving the way for more intelligent and adaptive software quality assurance methodologies.

---

Keywords

---