Transfer Learning-Based Semi-Supervised Generative Adversarial Network for Malaria Classification

1  Introduction

Malaria is one of the most common causes of mortality in the world. Every year, it affects millions of people and leads to millions of deaths. According to WHO, nearly 228 million active malarial cases were reported in 2018 alone, and there were an approximate 405000 deaths worldwide [1]. Most of the malaria cases were reported from African regions (213 million or 93%). These are poverty-hit countries with very few resources to battle malaria. Traditional methods of diagnosing malaria, such as microscopy, rapid immune-chromatographic tests (RDTs), and polymerase chain reaction (PCR) [2] are not efficient enough to detect malaria. Malaria diagnosis and detection should be fast and accurate to prevent further spread in the community. Based on traditional Machine Learning (ML) and state-of-the-art deep learning (DL), numerous automated diagnosis systems for malaria have been presented to overcome the limitations of the conventional diagnosis methods [3]. However, it is still needed to improve these systems in terms of performance, training time, and amount of training data.

Traditional machine learning approaches are limited in their ability to interpret images in their raw form [4] and require extensive image preprocessing and segmentation processes. Moreover, the informative features are mostly extracted by experts based on their knowledge about malaria. Non-experts find it difficult to utilize ML approaches for their studies due to expert judgement and a time-consuming trial and error technique in the feature selection process [5]. By including the feature extraction stage into the process of learning, DL resolves this difficulty. In contrast to conventional machine learning techniques, the key features are extracted automatically in a self-instructed fashion [6], as a result, the responsibility of extracting features transferred from humans to computers, allowing non-experts to explore DL more effectively for their research. The amount of data, on the other hand, has a big impact on the potential of DL-based diagnosis systems. To prevent over-fitting, such models need a significant amount of data for training [7]. Moreover, due to expensive data collection and labeling processes, there is a scarcity of large labeled datasets in the medical field [4,8]. These challenges usually limit the use of DL models in building diagnosis systems.

TL has mostly been adopted in the literature to handle limited medical data [9]. Numerous diseases such as brain tumors [10], diabetes mellitus [11], breast cancer [12], Covid-19 [13], and Alzheimer [14] have been automatically diagnosed using pre-trained DL models. TL has also been used in combination with generative adversarial network (GAN) for the classification of limited medical data [15]. Integration of these two techniques also helped reduce the training time and enhance the quality of generated images [16]. Furthermore, it has been demonstrated that with less labelled data, the semi-supervised GAN (SGAN) can achieve performance comparable to a traditional supervised Convolutional Neural Net-work (CNN) for medical images [17]. Motivated by the existing research, we propose a model in this paper (which is an extension of our conference paper [18] ) to detect malaria, which is based on transfer learning (TL) and SGAN (TL-SGAN). Unlike a conventional CNN model, the proposed TL-SGAN is trained with fewer data. Another important aspect of the model is that only a small fraction of the training data is labeled. The main contributions of the current research work are as follows:

•   The two concepts mentioned above, namely SGAN and TL are combined into a single model. The proposed model demonstrated excellent ability to detect malaria in blood cell images.

•   The proposed TL-SGAN is trained in a semi-supervised manner using a small number of images. This approach makes the diagnosis and treatment of malaria more convenient and can relieve pathologists of enormous workload.

This research is an extension of our research work published in [18]. Rest of the paper is organized as follows. Literature review and background study related to this research work are given Section 2 and 3 followed by the detail explanation of the proposed methodology in Section 4. Empirical results are discussed in Section 5. The research work is concluded in Section 6.

2  Literature Review

Traditional machine learning methods have been widely applied to create automated diagnostic systems for malaria in recent years. In [19], an effective malaria detection technique is presented, where the erythrocytes were segmented from microscopic images using watershed transformation. After feature selection, support vector machine (SVM) and Bayesian network were utilized for classification. The Bayesian network attained an accuracy of 84.0% using the 19 most essential features. A generalized variation denoising method and a local adaptive threshold approach were utilized for noise removal and segmentation, respectively, in an image-based cytometer for quick malaria detection [20]. After that, SVM was used to classify the segmented images, with a minor false positive rate of 0.0025%. P. falciparum and white blood cells were identified in blood smear pictures using a mobile device-based automated approach [21]. Blood cells were segmented using adaptive threshold and then classified using SVM. Two image processing systems for the detection as well as the classification of malaria were proposed in [22]. A morphological algorithm was used was used to detect malaria in negative binary images with 95% accuracy, whereas 100% accuracy is observed while classifying the parasite into four species using normalized cross-correlation function. For the automatic detection and staging of RBCs infected with P. falciparum, quantitative phase analysis of unstained cell images was used [23].

DL has also shown enormous success in the automated diagnosis of malaria. A 19-layer CNN was demonstrated for classifying infected and uninfected malarial cells, with an accuracy rate of 98.9% [24]. The use of multi-wavelength imaging to aid in data augmentation was used in a quick and robust malaria classification system [25]. A CNN model with five convolutional layers and two fully connected (FC) layers was used to achieve a classification accuracy of 97%. Various combinations of different numbers of layers and nodes were used to identify malaria using a deep belief network [26]. The network’s optimal number of layers and nodes yielded a 96.21% accuracy. Faster R-CNN and Alex-Net for object detection and classification were used in a two-stage malaria detection and classification model that achieved a 98% accuracy [27]. VGG and SVM were used to identify falciparum malaria with 93.1% by substituting the last layers of VGG with SVM [28]. Various object detection pre-trained models such as Faster R-CNN, Single-shot multi-box detector, and RetinaNet were utilised for malaria parasite detection. The results demonstrated that Faster R-CNN is the most well-trained model with an average precision of over 0.94 [29]. For automatic diagnosis of malaria, stacked CNN [30,31] and an ensemble of residual networks [32] were recently used, with accuracy rate of 99.96% and 98.08%, respectively.

3  Background Study

This section briefly discusses the background of the material and methods used in this research.

3.1 Malaria

Plasmodium, a single-celled parasite, causes malaria, which is an infectious disease. When female Anopheles mosquitos bite humans and feed on their blood, they absorb the parasite and disseminate it. P. falciparum, P. vivax, P. malariae, P. ovale, and P. knowlesi are the five Plasmodium species that cause malaria in humans [33]. P. falciparum and P. vivax are the most abundant of the five kinds. The Plasmodium life cycle comprises two phases: sexual, which happens in mosquitoes, and asexual, which occurs in humans. A Plasmodium parasite enters the human body as a sporozoite and then enters hepatocytes, where it undergoes its first phase of asexual growth. Red blood cells (RBCs) go through a second period of asexual development, which eventually leads to their bursting [34]. Malaria infection symptoms might include headaches, nausea, vomiting, photophobia, and muscle aches in the early stages.

3.2 Transfer Learning

TL is a machine learning technique that retains and uses information obtained from addressing one problem to a related problem [8]. The idea is to apply a model’s understanding (e.g., ImageNet) to solve a different problem (e.g., Malaria detection) with a smaller data set is shown in Fig. 1. In TL, rather than learning from scratch, previously acquired patterns are applied to a new problem. TL is usually accomplished using DL models that are pre-trained on huge and demanding object recognition tasks such as ImageNet. Pre-trained models are either used as feature extractors or adapted to the new problem by fine-tuning their last layers. Reduced training time, increased efficiency, and the ability to work with a smaller data set are just a few of the benefits of TL [9].

Figure 1: Transfer learning

3.3 VGG16

VGG stands for the Oxford University’s Visual Geometric Group, and VGG16 is a deep learning model first presented by this group during the ImageNet challenge in 2014 [35]. It consists of thirteen convolutional layers and three FC layers. The model receives an input of size 224 × 224 × 3, which is convolved using kernels of size 3 × 3 (with stride and padding set to one pixel). The convolution operations in VGG16 do not change the spatial dimension of the feature maps. Moreover, five spatial max-pooling operations are performed over a 2 × 2-pixel window (stride size set to 2), and as a result, the spatial dimension of the feature maps becomes half. Two FC layers (each with 4096 nodes) are present between the convolutional stack and the output layer. The hidden layers in the network are equipped with rectified linear unit (RELU) activation function, while the output layer is a softmax layer consisting of 1000 nodes and classifies the images in the ImageNet data-set. The architecture of VGG16 is given in Fig. 2.

Figure 2: VGG16

3.4 SGAN

SGAN shown in Fig. 3 is one of the most exciting applications of GANs in practice [36]. SGAN requires only a small fraction of the training data to be labelled, as compared to supervised learning approaches that require all of the data to be annotated. A generator and a discriminator are the two deep neural networks that constitute SGAN. The generator takes a random vector (z) as an input and outputs fake graphics (x*). Real images without labels from the training dataset (x), false images made by the generator (x*), and real images with labels from the training dataset (x, y), where y refers to the label for the provided image x are all inputs to the discriminator. For each example, the discriminator calculates and gives the probability of an image being real. The discriminator utilizes fake images as additional information to learn patterns in the data. During training, it learns to differentiate between fake images and actual unlabeled images and classify labeled images in the training data into their corresponding classes. In a nutshell, SGAN is a classifier with C + 1 output nodes, where C is the number of classes present in the data. Fig. 3 depicts architecture of SGAN.

Figure 3: SGAN

4  Proposed Methodology

Fig. 4 shows the flow diagram of the proposed TL-SGAN, while the different steps involved are discussed below.

Figure 4: Flow diagram of the proposed TL-SGAN

4.1 Data Description

A publicly available dataset of 27,558 blood cell pictures with the same amount of infected and uninfected cells is used [37]. Plasmodium was present in the infected blood cells, but not in the uninfected ones as shown in Fig. 5. At Chittagong Medical College Hospital in Bangladesh, Giemsa-stained thin blood smear slides from 150 Plasmodium falciparum-infected and 50 healthy persons were investigated for image collection. In order to capture the photos, a normal Android phone was connected to a light microscope which were then manually tagged. The RBCs were segmented using a level-set based algorithm [36].

Figure 5: (a) Uninfected cells (b) Infected cells

4.2 Data Preparation

In this research, only 2500 images from the dataset are utilized. For each class, i.e., infected and non-infected cells, 1250 images are randomly selected. The images are initially reshaped to attain same dimension of 128 × 128 × 3 and then normalized. The normalization is performed by dividing each pixel value in the images by 255. The data is then divided into training (1500 images) and testing datasets (1000 images), respectively. Moreover, 500 images in the training dataset are labeled, while the rest (1000) are unlabeled. Furthermore, the testing dataset is divided into five different sets each having 200 images as shown in Table 1. It also summarizes the distribution of images into different datasets.

4.3 Model Architecture

The discriminator in SGAN is replaced by a pre-trained VGG16 model in the proposed TL-SGAN. The generator is a CNN with an FC layer that accepts a size 100 noise vector (z). The input vector is reshaped into a four-dimensional tensor and used as the starting point for the generator’s convolutional stack (six layers). The input is UpSampled before each convolutional layer, thus doubling its size. Similarly, except for the last convolutional layer, all of the convolutional layers utilize Relu as the activation function. The last layer makes use of the Tanh activation function to generate bogus images that are 128 × 128 × 3 in size. A few changes are made to the VGG16 (discriminator) architecture to make it more suitable for the problem at hand. Based on the image input size, the size of the convolutional and pooling layers is scaled. A single global average pooling (GAP) layer replaces the FC layers, calculating the mean of each feature map. A GAP layer contains fewer parameters than FC layers, which reduces the risk of over-fitting and the computational burden [7]. Similarly, the VGG16 replaces the single softmax layer (with 1000 nodes) with two output layers (each with two nodes), one to classify infected and uninfected blood cells, and the other distinguishing between fake and real photos. To put it another way, the proposed TL-SGAN architecture is a single architecture having two output layers that use similar feature selection layers. Fig. 6 depicts architecture of the TL-SGAN.

Figure 6: Network architecture

The generator and one output layer (which differentiates real images from fake) in the discriminator are entirely disregarded after training. The rest of the discriminator is retained and is used to classify the unseen data into infected and uninfected cells. The proposed model’s classification performance is evaluated using various statistical measures and is compared with the existing models.

4.4 Training Process

In the TL-SGAN, both the label and unlabeled datasets are taken and are utilized to train a generator model, a supervised and an unsupervised discriminator network, simultaneously. Such a model can predict C + 1 classes using a softmax activation function. Algorithm 1 represents the training process for the TL-SGAN. During SGAN training, random noise is introduced to the unlabeled (x) and labeled (x,y) real images so as to generate fake images. Where y is the labels associated to x. The fake images are then input to the discriminator for predicting whether they are real or fake. Parameters of the generator are updated iteratively till it predicts the fake image as real.

4.5 Model Evaluation

At the end, the proposed model’s classification performance is tested using multiple performance measures and compared to current models.

5  Results and Discussion

5.1 Experiment Setup

The experiments in this study are implemented on Tensorflow, an open-source deep learning system. Google Colab that is a cloud-based service with free GPU support is use to run the code. During training, the discriminator’s last three layers are fine-tuned, while the remainder of the first layers are frozen. On the malaria dataset, model training took roughly 500 iterations. With adam as the optimizer and binary cross-entropy as the loss function, a batch size of 64 is employed.

5.2 Evaluation Matrices

Statistical metrics such as precision, accuracy, sensitivity, F1 score, specificity and area under curve (AUC)-receiver operating curve (ROC) are calculated to measure the classification performance of the TL-SGAN model on testing dataset. These metrics depend on four main statistical values, such as true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), associated with the classification model. In this research, TP indicates the number of correctly categorized infected cells, while TN represents the number of healthy cells classified without error. FP and FN, on the other hand, represent misclassified infected and uninfected cells, respectively.

The Accuracy metric estimates the ratio of correctly classified images to the total images as in Eq. (1).

Accuracy=(TP+TN)(TP+TF+FP+FN)(1)

Precision metric determines the amount of correctly predicted infected cells to all the infected cells that the model predicts and can be expressed as in Eq. (2).

Precision=(TP)(TP+FP)(2)

Sensitivity, also referred to as recall, calculates the number of infected cells that the model predicts correctly as in Eq. (3).

Sensitivity=(TP)(TP+FN)(3)

The specificity metric calculates the number of those normal cells that are rightly identified by the model and can be obtained using Eq. (4).

Specificity=(TN)(TN+FP)(4)

F1 score, also known as F-measure, measures the weighted average of precision and recall as in Eq. (5).

F1=2∗(Precision∗Recall)(Precision+Recall)(5)

ROC plots two parameters i.e., true positive rate (calculated using Eq. (6)) and false positive rate (obtained using Eq. (7)). AUC value estimates the area under ROC curve and reflects the capability of our model to differentiate between the two classes.

True Positive Rate=TP(TP+FN)(6)

False Positive Rate=FP(FP+TN)=1−Specificity(7)

5.3 Result Discussion

As discussed in subsection 4.2, the TL-SGAN model uses five test sets to evaluate its performance. Fig. 7 shows the confusion matrices, while Table 2 presents the average values of the statistical measures achieved by the model for five testing datasets. A high recall value means the model has a low number of false negatives and can accurately identify parasitic cells (Fig. 7). Precision, on the other hand, is slightly reduced as a consequence of the greater number of false positives. The number of false positives is greater because the dataset contains some non-parasitic images that are difficult to separate from infected cell images, causing the model to misclassify them. Fig. 8 displays a few examples of uninfected cell pictures incorrectly categorized as infected by the model. Similarly, a high AUC score of 0.98 is attained by the proposed TL-SGAN (Fig. 9), which means the model has a high magnitude of separability for infected and uninfected cell images.

Figure 7: Confusion matrices: (a) Dateset-1 (b) Dateset-2 (c) Dateset-3 (d) Dateset-4 (e) Dateset-5

Figure 8: Misclassified uninfected cell images

Figure 9: AUC-ROC

5.4 Comparison

The proposed model is evaluated against some of the recent models in Table 3 that have utilized the same malaria dataset. Except the proposed model, all other models have used a large training data and have followed a supervised approach. Our TL-SGAN model with a small labeled data for training has achieved better results as compared to other models. Since it used a huge quantity of data for model training, the Squeeze and Excitation Network (SENet) has somewhat better accuracy (97%) than the proposed model [38]. They nearly tripled (82674 images) the size of the original dataset (27558 photos). SENet for malaria detection is computationally expensive and requires more training time than the suggested model because it uses such a huge number of images. The SGAN was trained from the ground up using the same number of photos as the proposed model. The SGAN trained from scratch not only took more time to train (as can be seen in Fig. 10) but also has led to over-fitting. This inference is based on the fact that SGAN (trained from scratch) achieved much higher training accuracy (98%) than the test accuracy (only 65%). In this case, the discriminator had a large number of trainable parameters against a small training data set, resulting in bad generalization performance.

Figure 10: Time comparison of SGAN and TL-SGAN on ten different trainings

The impact on accuracy by changing the number of labeled images is also investigated. For this purpose, different numbers of labeled images (multiple of 50 up to 500) are used to train the model and then evaluated on the same testing dataset. The graph in Fig. 11 plots the number of labeled images against the mean accuracy of the model on five testing datasets. It was observed that even with 50 labeled images, accuracy of 81.34% is achieved, which is already acceptable for the given size of training data. As the number of labeled images is increased, the accuracy improved further. Initially, the slope of the accuracy curve is steep, slowing down and finally leveling off. Eventually, a peak accuracy of 96.6% is obtained with 500 labeled images.

Figure 11: Labeled images vs. accuracy

6  Conclusions and Future Work

This research aims to solve major medical image analysis issues, particularly in the automatic diagnosis of malaria. Malaria is automatically detected using TL and SGAN by classifying blood cell photos into infected and uninfected classes. Because the proposed model does not involve manual feature extraction and segmentation of microscopic images, it outperforms standard ML-based models for malaria detection. The deep CNN network (VGG16)’s convolutional and other hidden layers are strong enough to automatically extract essential features and distinguish infected cells from uninfected cells. DL models typically need a large amount of label data for training, which is very difficult to collect, particularly in the medical field. In this research paper, despite using small training data, accuracy of 96.6% is obtained for malaria detection. Similarly, the model is trained in a semi-supervised manner by annotating only a small portion of the training dataset. This semi-supervised approach is less depended on the medical experts and can overcome the problem of annotating all the images.

Our work is limited to the classification of healthy and Plasmodium falciparum-infected cells. In future work, the diagnosis of malaria and its treatment can be made even more effective by extending this research work to the detection and classification of malaria parasites into sub-classes. The identification of sub-types of Plasmodium will help the physicians in recommending a particular medicine and treatment plan. Moreover, the proposed methodology ability may be investigated on multi-class classification problems as well. Similarly, the VGG16 (discriminator) in the proposed model can be replaced by other pre-trained models such as ResNet, Inception, DenseNet, etc., successively. The diagnosis of malaria by the proposed model would improve if any other pre-trained model performs better than VGG16. The medical data can be noisy and can harm the performance of the model. The proposed methodology has not used any noise removal method as the data used in this research work does not have a high noise.

Author Contributions: The published version of the manuscript has been read and approved by all authors.

Funding Statement: The publication of this article is funded by the Qatar National Library. The authors would like to acknowledge the library for supporting the publication of this article.

Conflicts of Interest: The authors declare no conflict of interest.