An Efficient Unsupervised Learning Approach for Detecting Anomaly in Cloud

1  Introduction

Cloud Computing (CC) is a rising Internet model that offers complete services and provides them with complex software, hardware and protocol stack functionality [1]. Users make use of these services for productive computing. Even though service usage is an essence for CC, it will not provide ineffective information. Moreover, it abuses and attacks the network architecture [2]. For instance, a spiteful user resides over Virtual Machine (VM), intrudes on other VMs over the cloud, and utilizes other machines to spread malicious software or Denial of Services (DoS) attacks. However, the user's character will produce huge network traffic specifically to traffic users’ accessing services from other networks and shows traffic among VM in the virtual network [3]. Cisco industries’ research reports that global network traffic accounts for 95% of the total traffic network traffic by 2022. Specifically, traffic among VMs in a cloud environment may account for 85%. This traffic may be consistent and increased drastically and inevitably encounters malicious attacks [4]. A network attack results in huge damage to the cloud; however, it causes users to lose confidence in CC, affecting CC's sustainable and healthy development [5]. Intrusion detection is a technology for protecting the CC from various malicious attacks. Hence, cloud-based intrusion detection identifies and examines network traffic, specifically identifying malicious attack characteristics and eliminating damage [6]. Therefore, guaranteeing reliability and safety towards CC.

Also, cloud infrastructure is modeled with virtualization technology that renders virtual flow among VMs uncontrollable and invisible by conventional IDS (See Fig. 1). It behaves as a global view, centralized control and programmability. Therefore, it is extensively utilized in CC. In diverse investigations like [7], the author anticipated using SDN technologies to redirect traffic to grunt IDS to identify attacks. Here, snort is used for the signature detection model, cannot identify unidentified attacks, and adopts traffic. Anomaly detection is used for distinguishing anomalous traffic from normal traffic [8]. It is extremely suited for unknown attack detection; however, it shows a false alarm rate. Various kinds of Machine Learning approaches are used extensively in IDS Neural Network, Decision Tree, Random Forest and Support Vector Machine. Moreover, these techniques may offer detection accuracy and unsatisfactory classification. Intrusion detection-based outcomes depend not only on classifier performance [9]; however on the performance of input data quality. Usually, network traffic includes feature reduction and high dimensionality that causes feature dimensionality disaster [10]. Hence, feature detection is specifically essential for effectual enhancement in supervised classifier performance. It comprises two kinds of approaches: feature extraction and feature selection. Feature subset selection functions effectually by eliminating redundant or relevant features; feature subset selection may offer the finest performance based on a certain objective function. Various investigations have reported that feature selection approaches eliminate the ‘dimensionality curse and attain superior detection performance over NIDS. Feature extraction-based mapping provides original higher dimensionality features to lower dimensionality features and produces new non-linear and linear combinations of original features. Recently, diverse investigators have demonstrated deep learning technology with resourceful IDS for feature extraction. It uses ML approaches to automatically extract effectual features from raw data and input them into the classifier for attack identification. The significant contributions of the research work are:

1.    Initially, the dataset should be acquired from the online accessible resource. This work includes the NSL-KDD dataset for threat prediction.

2.    Next, the most influencing features need to be analyzed from the dataset as this threat may show a huge impact on the application. Here, an auto-encoder is used for feature selection.

3.    Finally, the prediction accuracy needs to be computed with the classifier model. Thus, the Naïve Bayes classifier is used for predicting the classifier's accuracy. The simulation is conducted in a MATLAB environment where metrics like accuracy, FAR, TPR are evaluated.

Figure 1: Generic IDS model

The remainder of the work is organized as. Section 2 depicts background studies of feature selection and classification approaches. Section 3 demonstrates anticipated auto-encoder-based feature selection and a Naïve Bayes classifier for modeling IDS. Section 4 illustrates numerical analysis and results of the anticipated model, and Section 5 gives results and directions for future research.

2  Related Works

This section discusses intrusion detection approaches that various other authors perform. An extensive study is done with IDS is discussed below. The author in [11] anticipated an anomaly detection approach that functions over the VM layer. These entries are stored in key-value format. Here, it specifies system cal, and values specify the list of system call. It executes program-based detection, cloud administrator and matches with database for identifying mismatch. It fails in longer traces, and attackers can evade those systems when VM's supervision process is done. The author in [12] depicts a frequency-based approach for safeguarding VM over cloud environment. Frequency counts of all system calls are maintained. Therefore, the system call order is lost. Approaches like the Markov model are utilized to generate more appropriate characteristics of malware-dependent probabilistic computation. The training time is extremely higher and shows substantial growth for huge databases. It includes experimental tasks and consumes time. It is diverse from other models as it converts frequency values to weights for performing testing/training with (k-Nearest Neighbor) k-NN classifiers.

The Virtual Machine (VM) model uses text mining techniques to generate feature vectors to enhance n-grams discriminative powers. It is utilized for feature selection by demonstrating the rarity and frequency of n-gram. Some models do not utilize virtualization features as introspection and hence lack predicting attacks that thwart IDS at VM. The author in [12] depicts VM based security framework for deployment over VM in virtualization. Policy engines are a core detection element that describes security for malicious activities by examining VM events. Signature-based approaches are utilized for detecting unknown and variant attacks. The author in [13] anticipated an IDS framework for modeling Operating System hooks. This hook is deployed over kernel modules to intercept OS events. Security analyzer examines the malicious activities of events. Also, attackers with kernel models may transmit false information. The author in [14] anticipates a VM-malicious process for detection. It uses a process for identification approach regarding the tracking process to handle address space. It is based on address space. Active IDS specifies some new processes. It resourceful executes the implementation of Lycosid based on ID implementation. Similarly, the classifier model is used for predicting anomaly ineffectual manner. Self-training is to train both unlabelled and labeled data. Initially, it produces classifier fitting for sample labeling. Then, the classifier allocates labels with unlabelled samples. It is retrained with both unlabelled and labeled data for prediction. This process is iterated recursively till it satisfies termination criteria. Hence, various efforts are used to develop unsupervised learning [15–17]. But it suffers from certain disadvantages where confident instances are self-labeled from decision boundaries. Training is trapped due to outliers. There are some spaces for self-training approaches. It fails to enhance the performance of random splitting and reduce data redundancy [18–20].

3  Methodology

This work uses CC to construct NIDS for decoupling conventional cloud infrastructure. This anomaly detection approach is utilized to acquire three essential functionalities. 1) Pre-processing where network-based traffic is standardized; 2) feature selection is performed with auto-encoders; and 3) classifier model. Finally, attack recognition is utilized to identify intrusion over network traffic and testing datasets. This section explains two modules and is discussed as given below.

3.1 Data Collection

It is considered to be a preliminary step for predicting intrusion. Here, a network collector is utilized for traffic monitoring from a pre-defined platform. The physical machine comprises domain name and non-privileged domain. It is connected with a conventional switch and VM for connection with a virtual switch. VM communicates with successive VM or over different PM through virtual switches. It is utilized for forwarding network flow, while the collector is accountable for network flow and routing control. It is attained to handle anomaly detection with intrusion detection. This pre-processing model comprises of transformation. Former is utilized to change nominal to numerical values. For instance, in the NSL-KDD dataset, three features are considered: Transmission Control Protocol (TCP) status flag, service type, and protocol type. It is determined to be more nominal. It transfers attack type to numerical values, i.e., 0–4 is normal, DoS, probe, Root to Local (R2L) and User to Root (U2R) specifically [20–22]. The description of the NSL-KDD dataset is given in Section 4. To eliminate feature bias with superior values and the huge amount of sparse features, 0 should be standardized for scaling every feature value to a proportioned range [23–25]. Here, Z-score is utilized for standardization.

3.2 Basic Auto-Encoder (AE)

Auto-encoders are considered a popular method in Neural networks towards the unsupervised fashion. This method is anticipated and examined. The ultimate target of this process is to learn the dense representation of inputs during the maintenance of some essential information. The basic autoencoder is generally a Back Propagation (BP) method based on original data at the input, which are reconstructed at the output while passing through the encoding layer by diminishing the number of hidden layers. AE learns compact and deep features with reduced hidden nodes. Therefore, this specification is considered to be entirely reconstructed with some original data. Fig. 2 depicts the flow diagram of the anticipated model.

Figure 2: Flow diagram of the proposed model

Consider, the training samples x(x = [x1, …, xm], xk ∈ Rm, k = (1, …, M) are the learning objective of an AE is to reduce the reconstruction error as specified in Eq. (1):

minimize:∑kM⁡||xk−xk^||2 (1)

Here, xk and xk^ are input and reconstructed outputs respectively. Mathematically, the hidden layer relies for encoding and decoding process as expressed in Eq. (2):

{Hf=G(af,bf,x)x^=G(an,bn,Hf) (2)

Here, af and a specify the weight for encoding and decoding layers, bf and bn are biased, and G(ai, bi, x) specify hidden neurons. The learning process is considered a preliminary auto-encoder that includes the iterative update of weights/coefficient s′ a′ and ′b,’ where the error among the inputs and reconstructed network outputs are gradually reduced until given below the threshold value. Generally, the learning processes of the auto-encoders are based on BP fine-tuning and layer-wise initialization. The anticipated AE structure is constructed in a multi-layer fashion. Initially, data transformation is performed to transfer input towards random feature space, followed by encoding layers to update the replacement learning process. At last, the parameters in encoding layers are updated, and input data is mapped into very low-dimensional features [26–28]. Mathematically, the two-layer network ′f′ based output data is expressed as in Eq. (3):

f=G(an,bn,G(af,bf,x)) (3)

Moreover, with multilayer network model, the output data for all hidden layers are expressed as in Eq. (4):

Hfi=G(afi,bpfi,Hfi−1) (4)

Here, Hi is ith layers output data, and Hi−1 is (the i − 1) th layer. There are diverse discriminations among the multi-layer auto-encoders. The multi-layer AE is completely different from prevailing BP-based AE, where the layers are merged to form a system. The system parameter needs to be iteratively retrained. The training speed of the anticipated model is multiple times faster than the conventional approaches. Similarly, the hidden nodes of the encoding layers are generated randomly. However, it is unreasonable to generate all the layers randomly. It is because the random layers can destroy the essential features. Therefore, the anticipated model with hidden nodes is evaluated and relatively nearer to the input data. However, the network layer needs a weighted output layer to absorb system errors as the hidden nodes are randomly generated. This model is naturally asymmetric as output layers’ weight is not required during the encoding process but prevails during the decoding process. The unnecessary mapping can eliminate the necessity of low-dimensionality features [29–32]. The anticipated model can remove the learning system naturally. The features are optimized based on the preliminary findings of optimal parameters to reduce the output error with top-priority AE.

3.3 Learning Rate

This section explains the training process of the network architecture. Consider, ′M′ training points {(xk,yk} k=1M,xkϵRm from the continuous system. The AE pretends to reconstruct the original input by x = y, y → (0, 1). The initial parameters of encoding layers are attained with the random process as expressed in Eq. (5):

Hf=G(af,bf,x)

(af)T.af=I

(bf)T.bf=1 (5)

Here, af ε RD*m is random weights and bias of encoding layers. Initially, bf ε RD, then bf belongs to ′R.’ Hf is feature data. The hidden neuron ′G′ is based on linear function, RBF function, sigmoid function, wavelet function, threshold function, and applied in decoding layer. Then, the invertible function (sine or sigmoid) can attain parameters from the encoding layer and be expressed as (6):

G−1(.)={arcsiin(.)ifG(.)=sin⁡(.)−log⁡(1(.)−1)ifG(.)=1/(1+e−(.)) (6)

Here, Moore inverse matrix H is used. With Moore's matrix and invertible function, the solution possesses the least norm between the least-square solutions. The projection techniques are efficiently utilized for evaluating the inverse matrix; H†=(HTH)−1HT; if HTH is non-singular; then H†=HT(HTH)−1 if HHT is singular. Based on regression theory, it is recommended to have a positive value and added to the diagonal of HHT. It enhanced the stability and is expressed as in Eqs. (7) and (8):

H†=HT(CI+HHT)−1 (7)

H†=(CI+HHT)−1HT (8)

Then, update af, bf and expressed as in Eqs. (9) and (10):

af=(an)T; (9)

bf=bn;bfϵR (10)

The feature data is updated with Hf = G(af, bf, x). the above-given process is repeated concurrently. The parameters an, bn, af, and bf are attained along with the feature data Hf. Unlike BP, the conventional AE uses network weight and establishes it by pulling hidden nodes’ input and output data. Moreover, when the parameters of encoding layers are randomly generated, the random encoding process can eliminate the essential information. There the input weights of the methods are evaluated and extremely correlated with input/output data. The anticipated model shows some benefits as the weight of the encoding layers does not compute and replace the weights of the decoding layer. Therefore, the computational workloads are eliminated. Based on this analysis, training error is reduced, and the testing process is increased gradually. Thus, the testing accuracy improves with the promising solution and diminishes enormous computational workloads. It is given in Algorithm 1:

3.4 Auto-Encoder for Feature Selection

Auto-encoder is an unsupervised dimensionality reduction approach that comprises of encoder and decoder [25]. It includes input, hidden and output layers. The encoder is cast-off for reducing dimensionality, and the decoder is utilized for reconstruction. The functionality is a vice-versa process. The training dataset has various’ i with feature reduction vector and class labels. Encoder functionality is mapped as input to hidden nodes, and decoder functionality is specified as the hidden specification for reconstruction [26]. When hidden layer neurons are smaller than several input/output neurons, the compression vector is attained and realizes dimensionality reduction. This process is depicted as in Eqs. (11) and (12):

h=f(wx+b) (11)

z=g(w′h+b′) (12)

Here, ′f′ and ′g′ are non-linear activation functions. ′ω′ and ′ω′ are weighted matrices, and ′b′ and’ b′ are biased values. Here, θ = {w, b} and θ′ = {w′, be} where θ′ and θ specifies encoder and decoder parameters. The ultimate objective is to reduce reconstruction error among input and output by handling these factors. The objective functions are depicted as in Eqs. (13) and (14):

Jauto−encoder=∑xϵDn⁡L(x,z) (13)

=∑i=1n⁡L(xi,gθ,(fθ(x(i))) (14)

Here, ′L′ is reconstruction error is MSE and cross-entropy loss. It is formulated as in Eqs. (15) and (16):

L(x,z)=1n∑i=1n⁡(x(i)−z(i))2^ (15)

L(x,z)=−1n∑i=1n⁡[x(i)log⁡Z(i)−(1−x(i))log⁡(1−z(i)) (16)

The smallest reconstruction error is depicted with output to input that implies effectual lower dimensionality feature specification. Here, reconstruction leads to an identical output problem to input. Thus, an auto-encoder is efficient in selecting features. To resolve this crisis, this work uses constraint specification or input corruption with the addition of noise. Unlike traditional AE, auto-encoder-based de-noising attempts to recognize effectual and appropriate feature specifications from noisy input data (See Fig. 3). It corrupts input and transmits that corrupted data to the anticipated mode for de-noising. Finally, it reconstructs the ′x′ format. It gives objective function as in Eq. (17):

J=∑i=1n⁡E_(x~(i)−q(x~(i)|x(i))L(x(i),gθ(fθ(x^(I)))) (17)

Figure 3: Auto-encoder for selecting appropriate threat features

The noisy inputs are attained from the corruption process. Some approaches include Gaussian noise and masking noise for setting random input features to zero.

3.5 Naïve Bayes Classifier

This model is a traditional approach with the Bayesian probabilistic model. It works on a stronger independence assumption towards probability measures. The probability measure does not influence the probability of other models. With a series of attributes 2n! gives independent assumptions. The outcomes of the NB classifier are generally more appropriate. The functionality of the NB classifier works effectually. It is depicted based on error factors like training noisy data, bias and variance. Training data noise is reduced by selecting appropriate training data. It is partitioned into various groups. Bias is known as an error owing to the training data group, which is extremely larger. Variance is depicted as an error owing to a grouping of smaller values. NB is a better classifier model when compared to existing approaches. It is because the essential characteristics of NB are extremely stronger assumptions towards independence from every event or condition. Next, it is easier for execution. It is executed for larger datasets. The basics of the NB theorem are expressed as in Eq. (18):

P(C|X)=P(C)|P(X|C)P(X) (18)

where X is attributes; ′C′ is class; P(C|X) is even probability with X which has been occurred; P(X|C) is even probability C which has been occurred; P(C) is probability of ′C′ event; P(X) is probability of ′X′ event. Similarly, X is expressed as in Eq. (19):

X=(x1,x2,…,xn) (19)

The connectivity among X and C are expressed as in Fig. 4. Then, substitute ‘X’, bayes formula is expressed as in Eq. (20):

P(C|x1,x2,…,xn)=P(C)P(x1,x2,…,xn|C)P(x1,x2,…,xn) (20)

Figure 4: Naïve Bayes classifier

Moreover, P(C|x1, x2, …, xn) as in Eq. (21):

P(C|x1,…xn)=P(C)P(x1,x2,…,xn|C)=P(C)P(x1|C)(x2,…,xn|C,x1)=P(C)P(x1|C)P(x2|C,x1)(x3…xn|C,x1,x2)=P(C)P(x1|C)P(x2|C,x1)P(x3|C,x1,x2)…P(xn|C,x1,x2,…,xn−1) (21)

From the above Eq. (21), it is known that there are some complex probability factors. It is extremely complex for analysis. Based on Eq. (21), there should be some assumptions that are free from one another. Therefore, the equation is expressed as in Eq. (22):

P(C|x1,x2,…,xn)=P(C)∏i=1n⁡P(xi|C) (22)

With NB classifier, the maximal probability of every class is expressed using hypothesis of maximum posteriori as in Eqs. (23) and (24):

Hmap=arg⁡maxP(C|x1,x2,…,xn) (23)

Hmap=arg⁡maxP(C)∏i=1n⁡P(xi|C) (24)

In the NB classifier, the prediction class needs to be determined. However, if ‘X’ attributes are quantitative, the probability value is extremely lesser than P(X|C), which cannot predict the Hmap values. Thus, other approaches like Gaussian distribution have to be used as in Eqs. (25) and (26):

P=(Xi=xi|C=cj) (25)

P=12πσijexp⁡(−(xi−μij)22σij2) (26)

Here, ′P′ is the opportunity; Xi is the ith attribute; xi is the attribute value; ′C′ is the value; Ci is the sub-class; μ is the attribute mean value; σ is the standard deviation.

4  Numerical Results

This work considers a MATLAB environment with 6 GB RAM and a 2.2 GHz Intel Core i7 processor. Here, the NSL-KDD dataset is used for dataset validation (See Tabs. 4 and 5). The metrics like accuracy are used for predicting attacks (See Tabs. 6 and 8). Here, attributes like duration, protocol_type, service, flag, source bytes, destination bytes, land, wrong fragments, and urgent are considered samples. UDP and TCP are the two protocol types considered. The classifier test is given for class predictions that are loaded from the input file. The robustness of this model is measured for training and testing. The outcomes are validated with improved feature selection and classifier while comparison with another model. Accuracy detection is attained with diverse experimentation and constant valuation. Generally, a confusion matrix is utilized to determine the performance of the classification algorithm, as shown in Tab. 2. The terminology is extremely confusing; however, it is simple and gives a better understanding (see Tab. 1).

Here, true and false specify whether the class is predicted correctly or not. Similarly, positive and negative specify class prediction to show whether the class is correct or not. With the confusion matrix, accuracy, precision, F1-measure and recall are predicted. Here, accuracy, false alarm rate and true positive rate are predicted. The formulas for these metrics are expressed below in Eqs. (27) to (29):

Accuracy=TP+TNTP+TN+FP+FN (27)

TPR=TPTP+FN (28)

FPR=FPFP+TN (29)

Tab. 3 depicts accuracy prediction with the NSL-KDD dataset. Figs. 5 and 6 depict graph representation of accurate prediction. Tab. 7 depicts the TPR and FPR comparison.

Figure 5: Accuracy prediction

Figure 6: Accuracy prediction with NSL-KDD

This work compares the detection accuracy of anticipated NB with other models like J48, AD tree, SVM, random forest, Bayes model and decision stump, respectively. The anticipated model acquires 93% accuracy for classifier performance. The other model attains 91%, 89%, 88%, 74%, 78% and 80% respectively.

From the Table mentioned above, accuracy is computed based on evaluation with the NSL-KDD dataset. The prediction accuracy of NB is higher compared to other approaches. The detection accuracy of the anticipated NB approach is given in the above Table. The maximal prediction accuracy is 93%, superior to fuzzy models, RF, RT, SVM, MLP, etc. It is known that the anticipated model is more appropriate for classifying intrusion detection. The NB model shows the finest and productive performance compared to other models using testing and training data. Intrusion detection is done more effectually and suited as a basis for classification for anomaly detection. False Alarm Rate (FAR) and True Positive Rate (TPR) are the metrics used for evaluation (See Figs. 7 and 8). The comparison is made among existing approaches like SVM, k-NN, J48, AB tree, RF, RT, MLP, and Fuzzy approaches. The FAR of the proposed NB is 0.3, which is lesser when compared to other approaches. Similarly, the TPR of NB is 0.99 which is 0.5%, 0.1%, 0.2%, 0.6%, 0.4%, 0.10%, 0.14%, 0.6% higher than SVM, k-NN, j48, AB tree, RF, RT, MLP, and Fuzzy respectively. The training of dataset samples are done with various percentages, i.e., 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, and 90% respectively with an accuracy rate of 91.02%, 90.50%, 90.50%, 91.30%, 90.56%, 92.86%, 93%, 91.25%, and 90.48% respectively. This work uses a 70:30 ratio for training and testing data partitioning. The auto-encoding-based feature selection can be identified for the NSL-KDD dataset. The anticipated model shows the finest algorithm within IDS with maximal intrusion detection accuracy. From Tab. 7, the FAR computation is measured to be 0.3 with 99% TRP. It is shown in Fig. 9, where the theoretical and experimental outcomes are determined. With this, it is known that the anticipated model works well in predicting the anomaly over the network.

Figure 7: FAR and TPR computation

Figure 8: Anomaly detection probability

Figure 9: Training accuracy rate

5  Conclusion

The summary is based on the proposed model while performing various experimentations using the NB algorithm for anomaly detection. In recent days, eliminating security breaches encountered with current technologies has been extremely unrealistic. Therefore, intrusion detection is essential for measuring features in network security. Moreover, detection techniques are not capable of identifying unknown attacks. Thus, anomaly detection has to be utilized for predicting attacks. In contrast, the Anomaly detection approach is used for enhancing intrusion detection accuracy. This work develops an improved NB model for anomaly and intrusion detection. This method is extremely effective for identifying various attacks and showed superior detection accuracy compared to other models. This model could classify data as abnormal or normal. It is utilized to predict accuracy and improve performance; IDS-based accuracy and performances of the anticipated model have been enhanced, and it is executed over real-time network environments. The experimental outcomes demonstrate that the anticipated approach attains superior accuracy with 93%, higher than J48, AB tree, RF, RT, MLP, SVM, and Fuzzy. Similarly, FAR and TPR of NB is 0.3 and 0.99, respectively. The limitations of this work are the systematic analysis of the constant dataset, but the research lacks real-time investigation of the threat. Similarly, zero-day attacks should also be analyzed as it shows a huge impact on real-time applications. In the future, a deep learning classifier or ensemble classifier will be used for enhancing prediction accuracy.

Funding Statement: The authors received no specific funding for this study.

Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.