Analysis of Security Testing Techniques

In the past decades, a significant increase has been observed in cyberattacks on the web-based systems used for financial purposes. Such individual systems often contain security weaknesses, called vulnerabilities that can be exploited for malicious purposes. The exploitation of such vulnerabilities can result in disclosure and manipulation of sensitive data as well as have destructive effects. To protect such systems, security testing is required on a periodic basis. Various detection and assessment techniques have been suggested by developers and researchers to address these security issues. In this paper, we survey the contributions of academia in the field of security testing for software applications and communication systems. A comprehensive review and in-depth analysis of the existing literature testing approaches has been performed to analyze their effectiveness and applicability under various scenarios. Further, we discuss various techniques used for conducting various security assessments. We follow the widely used method by Kitchenham and Charters for conducting a comprehensive systematic literature review process. Also, we propose a taxonomy for security testing techniques consisting of three main categories (Identification, Testing, and Reporting) and 17 subcategories consisting of specific security testing techniques (e.g., Black-box testing, risk assessment). Further, we assign a distinctive category from our taxonomy to each published paper in the security testing area, based on the material presented/discussed in the paper.


Introduction
A fundamental part of the Software Development Life Cycle (SDLC) is software testing comprising of two attributes, functional and non-functional. Functional testing is performed to verify the main functions/ requirements of the System Under Test (SUT). Non-functional testing verifies the non-functional aspects of the SUT, such as performance, usability, and security. Although security is a non-functional requirement, it is still considered one of the most important provision requiring an allocation of significant effort during the software development process as to ensure unhinged protection of the SUT. Software systems often contain weaknesses that make them vulnerable to various types of attacks. These weak points, also known as vulnerabilities, must therefore be identified, and appropriate security measures should be adopted to protect them from malicious incursions. One of the widely used techniques for the detection of such vulnerabilities is security testing.
Computer systems and applications are being affected by many vulnerabilities discovered from time to time. Such vulnerabilities may be exploited by attackers to disclose, modify, or delete confidential/private data [1]. They use various methods for exploiting vulnerabilities, for example, spoofing, man-in-themiddle attacks [2]. Several researchers have worked on developing new security testing techniques and improving the existing ones for the detection and prevention of vulnerabilities. Such research works are usually summarized in the form of Systematic Literature Reviews (SLRs). An SLR follows a pre-defined guideline to review the literature published in a specific field of study. In the field of security testing, very few SLRs have been carried out over the years that focus on specific vulnerabilities, applications, and hardware. Further, these SLRs discuss the literature focused on a specific platform only. One of the previously published relevant SLRs is the study by Latif et al. [3] that provides a review of various challenges, risks in cloud computing, and current countermeasures available to overcome them. Similarly, two other studies have been conducted by Dougan et al. [4] and Rafique et al. [5] in security testing. The authors provide a taxonomy based on the industrial and academic security testing process to categorize the academic literature. These studies are discussed in more detail in Section 3: Related Work.
In this study, we followed the standard methodology of Kitchenham et al. [6] for conducting the SLR. In addition, our contributions include: (i) we surveyed the most recent literature related to security testing techniques, (ii) we performed an in-depth analysis of each technique proposed in the literature for its effectiveness and applicability in various scenario, (iii) we proposed a taxonomy for security testing techniques and classified all our studied materials in appropriate categories.
The remaining paper is organized as follows: Section 2 provides the Review Plan for carrying out our study based on the guidelines of Kitchenham and Charters. In Section 3, we discuss the previous SLRs carried out in the field of security testing concerning our study. In Section 4, we present our proposed taxonomy for security testing techniques. Section 5 presents the research questions that we aim to answer in this study. The results have been discussed in Section 6 in detail. Finally, the conclusion is provided in Section 7.

Review Plan
This section presents the main components used in performing a Systematic Literature Review based on the guidelines of Kitchenham et al. [6]. It provides an overall summary of the information related to the collection and analysis of the relevant literature for our study.

Academic Search Engines
For this study, we collected the published papers, from various academic journals and conferences, related to Computer Science specifically focused on cybersecurity. A total of 292 papers were selected from the online portals via Google Scholar as listed in Tab. 1.

Search Strings
Tab. 2 lists the Strings and Keywords used for searching of papers during the study.

Inclusion Criteria
Following is the inclusion criteria for the selection of papers for this study: Papers are published in the period 2010-2019. Papers provide information related to security testing. Papers propose methodologies and techniques used for security testing.

Exclusion Criteria
The rejection criteria for papers is given below: Other sources Multiple Sources 48 Cyber risk assessment Papers that do not contain any information related to security testing are excluded. Papers published before 2010 are excluded.

Quality Checking Criteria
For quality assessment, the following points are considered regarding each paper: Numbers of citations of the specified paper/study. Quality/Impact Factor of the journal/conference where the paper has been published/presented.

Related Work
Rafique et al. [5] carried out an SLR to analyze academic literature related to the detection of security vulnerabilities and their effects on web applications. The authors' criteria for selection is based on integrating the security methods in software development lifecycle, mostly in the requirements, design and implementation phases, with papers published in the period of 2002-2015. They follow the guidelines of Kitchenham et al. [6] for conducting an SLR. Their findings show the lack of standardized techniques for security testing.
De Franco Rosa et al. [7] presented a review on the use of ontologies in security assessment. The authors discuss their findings about the papers on security assessment, the number of citations, and important concepts of each paper. The domain of this study is limited to system security and software testing with a focus on top-level ontologies, tasks, and application ontologies. In contrast to security assessment, our study is focused on security testing. Another review paper, focused on the combination of risk analysis and security testing techniques for software security, is provided by Erdogan et al. [8]. The results show a lack of tool support as well as formal definition and empirical evidence. Similarly, Doğan et al. [4] performed a systematic literature review on web application testing techniques. The authors claim that web application testing is important because of its massive use as well as different programming languages of these applications. This study includes the papers published in the period of 2000-2013. Another review of literature on software testing techniques has been provided by Jamil et al. [9]. The authors discuss the depth and type of testing required at various phases of the software development and software release life cycle among the existing test methodologies.
Jaisawal et al. [10] discuss the issues and challenges of testing web-based systems. The authors summarize the previously published literature related to security testing including important results, e.g., various vulnerabilities affecting web applications, challenges faced by security testers, use of risk analysis in security testing, and use of agile techniques in security testing. Garousi et al. [11] conducted a survey of previous reviews published in the field of software testing. The study is focused on identifying the challenges in software testing that have been investigated in the past. The authors claim that the survey papers received more citations than the systematic mapping/literature review papers. Similarly, Bertoglio et al. [12] performed a study on the papers published regarding penetration testing. They performed a systematic mapping of various studies published related to penetration testing. The study is structured using the PICO (Population, Intervention, Comparison, Outcome) technique.
In contrast to all of the above studies, our work encompasses the security testing techniques used in the detection of vulnerabilities and their classification based on existing standards. In addition, our proposed new taxonomy and analysis of a large number of latest papers (in the period of 2010-2019) are the unique features of our study.

Proposed Taxonomy
Based on the analysis of the existing literature and industrial standards, we propose a generalized taxonomy to bridge the existing gap between the academic studies and industrial literature [13][14][15][16]. We have also used this taxonomy to classify the large number of papers that we studied in this work. Fig. 1 depicts the proposed taxonomy where various security testing techniques have been categorized in three levels (1-3) based on the types of activities in each technique. Following is a short description of each security testing technique:

Identification
The identification category includes the techniques that utilize enumeration and reconnaissance of an application or a system, and identify the system's vulnerabilities/threats. These techniques utilize publicly available information on vulnerabilities from different CVE, Exploit-DB, etc. All such techniques are mostly referred to as vulnerability scanning or vulnerability discovery.

Testing
We classify the testing techniques into various subcategories based upon their interaction with the system, time of processing, and knowledge requirements. Most of the existing techniques are classified into three main categories: Black-box, White-box, and Gray-box.

Black-box Techniques
Black-box testing includes the techniques where information on the internal working of the application is not known. Such techniques examine the fundamental aspects of the system, having little relevance to the internal structure. They can further be divided into the following two subcategories: Fuzzy Testing: A testing technique that utilizes the execution of randomly generated data to infect or damage a system. Property-Based Testing: A type of model-based approach focusing on the conversion of security properties into specifications. It extracts the code relative to the specific property using program slicing.

White-box Techniques
White-box testing techniques are used in the detailed investigation of the software. In such a technique, the tester is provided with all knowledge of the system under test including the source code.
Model-based: This technique generates system models to check their adherence to the security requirements and properties. Static Analysis: Static Analysis, also termed manual code review, is a technique for finding vulnerabilities in the source code. Fault Injection: The fault injection technique tests the interaction points in a system. It further tests the limits of how far a system can be forced to execute malicious commands. Risk-based: These testing techniques are a subgroup of the model-based approach and utilize the previously solved test cases to mitigate any potential risks to the system.

Gray-box Techniques
Gray-box testing techniques are used for testing the application/system with limited knowledge of the internal structure. It increases the testing exposure by focusing on all layers of the system by combining the white-box and black-box techniques.

Other Testing Techniques
This category includes the techniques that could not be classified into any of the above-mentioned categories. They include: Penetration Testing: In the penetration testing technique, the tester attempts to find vulnerabilities in a system by mimicking the actual attacker's behavior to infect or damage the system. Ethical Hacking: Ethical Hacking comprises using penetration testing along with other hacking techniques to access a system in an unauthorized manner. While penetration testing involves the testing of vulnerabilities in the system, ethical hacking focuses on finding weak points in the whole environment including the clients and workers that interact with the system. Security Regression Testing: These techniques are used to identify vulnerabilities caused due to changes or up-gradation to the system and its functionalities.

Risk Assessment
Risk Assessment techniques are used to analyze the risks involved in the malicious use of the system and the potential damages. This technique utilizes various instruments for assessment, e.g., questionnaires, discussions, interviews.

Security Auditing
In Security Auditing, information systems are assessed by determining how well they follow the established guidelines.

Posture Assessment
Poster Assessment is a method for measuring and analyzing the overall security posture of an organization. It is a combination of security testing, vulnerability scanning, and risk assessment.

Research Questions
We investigate the following research questions in this paper.

RQ-1
How is the distribution of papers in security testing categories (Level 1)?
In this research question, we analyze the distribution of papers in the three main categories (Level 1) of our taxonomy, i.e., Identification, Testing, and Reporting.

RQ-2
What is the year-wise research publication trend in each security testing category (Level 1) over the last 10 years? RQ-3 What is the importance of each security testing technique (Level 2)?

Results & Discussion
This section discusses the results of our analysis during the systematic literature review. Tab. 3 lists the papers that were collected in our initial data set and the assigned categories in Levels 1 and 2 of our proposed taxonomy for each paper. The results are presented in the form of answers to our formulated research questions. To ascertain the use of each security testing technique, we have thoroughly analyzed the published work. Although we have 3 major categories in our proposed taxonomy, we also consider the survey papers as they constitute a good portion of our total collected papers.    As previously discussed, papers classified in the identification category make up approximately 50% of our dataset. For further analysis, we also looked at the year-wise distribution of these papers in each technique as shown in Fig. 4. As expected, the number of papers in the Identification category is more than any of the other categories in each year except 2014 where an equal number of papers were published in Identification and Testing categories. This further shows that the Testing techniques received more attention from researchers in 2014 compared to the Reporting and Survey categories.

Conclusion
In this paper, we have presented a comprehensive Systematic Literature Review (SLR) of the security testing techniques. The SLR has been carried out using the principles/guidelines of the widely used method of Kitchenham and Charters. Our dataset consists of the papers published in the recent ten years (2010-2019). We found a total of 292 papers relevant to Security Testing and thoroughly analyzed and discussed these papers from various aspects.
We proposed a new taxonomy for the classification of security testing techniques which consists of three major categories at level 1 (Identification, Testing, and Reporting), 8 security testing techniques at level 2, and 9 categories at level 3, e.g., Black-box testing, risk assessment. We assigned each studied paper to one of these categories based on the methods proposed/discussed in the paper. Our results demonstrate that, over the last 10 years, the highest number of papers and citations belong to the vulnerability scanning category. In addition, the papers published in the last 2-3 years have a very limited number of citations. Moreover, we also found a trend of using the older studies/techniques for security testing despite the fact that new studies have been conducted on yearly basis. Researchers and developers are still adapting the leadingedge techniques for the detection of new vulnerabilities. This research work is beneficial for future researchers of security testing as it provides a comprehensive analysis of the state-of-the-art in the field of security testing.

Conflicts of Interest:
The authors declare that they have no conflicts of interest to report regarding the present study.