An intelligent mosquito net employing deep learning has been one of the hotspots in the field of Internet of Things as it can reduce significantly the spread of pathogens carried by mosquitoes, and help people live well in mosquito-infested areas. In this study, we propose an intelligent mosquito net that can produce and transmit data through the Internet of Medical Things. In our method, decision-making is controlled by a deep learning model, and the proposed method uses infrared sensors and an array of pressure sensors to collect data. Moreover the ZigBee protocol is used to transmit the pressure map which is formed by pressure sensors with the deep learning perception model, determining automatically the intention of the user to open or close the mosquito net. We used optical flow to extract pressure map features, and they were fed to a 3-dimensional convolutional neural network (3D-CNN) classification model subsequently. We achieved the expected results using a nested cross-validation method to evaluate our model. Deep learning has better adaptability than the traditional methods and also has better anti-interference by the different bodies of users. This research has the potential to be used in intelligent medical protection and large-scale sensor array perception of the environment.
The disease transmission caused by mosquito bites has been a severe problem. The World Health Organization reports that mosquitoes kill 725,000 people a year, which makes them the deadliest insect in the world. Today, people in many areas of the world are still suffering from mosquito bites and mosquito-borne diseases. It is essential to reduce the risk of mosquito-borne infections through using mosquito nets for the people in mosquito-infested areas. However, there are some inconveniences if people use a conventional fabric mosquito net. In addition, when people rest in bed, the effect of blocking mosquitoes is reduced significantly if the users leave the net open to avoid the trouble of getting in or out of bed. Mosquitoes may have hidden in the net during the opening process.
Deep learning, which is a branch of machine learning, provides an excellent decision model tool for the control of Internet of Medical Things applications. Still deep learning is a technique based on artificial neural networks for the representation of data in machine learning [
We propose an intelligent mosquito net system which uses ZigBee and deep learning to solve the problem of perception of the users’ intention and the control of the mosquito net, and use the infrared sensor and the pressure sensor arrays to detect the status of the users. Next, we use ZigBee technology to process the data from the sensor, and transfer the data to the deep learning model for the classification of intentional status of the users. The model gives the intention of the current state of the users and controls the automatic opening and closing of the mosquito net subsequently. The infrared sensor detects the arrival of the users when the users enter the mosquito net, and open the mosquito net automatically. When the users need to leave the closed mosquito net, the model judges the intention of the users after the pressure sensor array detects the status of the users, and the key to the system is the intention of the users to leave the net.
The sensor will also detect the changes similar to the state of leaving the mosquito net, due to the changes of the body state during the regular rest of the users. In these situations, the users may be exposed to mosquito bites. In this case, we use the Dense Inverse Search (DIS) optical flow method [
The main contributions of this research are as follows:
A pressure sensor array is introduced to detect the status of the users in the process of entering or leaving the mosquito net; ZigBee is used in a mosquito net to network and control sensors with deep learning methods; The real-time feature extraction method and the DIS optical flow algorithm are introduced to extract the features of the pressure array; Both the three-dimensional–CNN method and the information of the time dimension are introduced into the attitude classification of the pressure array.
The article is structured as follows. Section 2 presents related work, and Section 3 introduces the workflow of the design. The detailed design is proposed, and we also present our experimental analysis in Section 4. Section 5 concludes the article.
Many researchers have made use of sensor devices on the bed for scientific research and practical applications. Gaddam et al. [
Su et al. [
Some researchers used bed sensors to detect falls [
Some researchers used bed sensors to analyze the stages or quality of human sleep[
Bed sensors for heartbeat detection have also been used by researchers [
Other researchers have focused on the detection of the human pulses or respiration [
More and more researchers are paying attention to the pattern recognition of the state of people in bed. In our system, the core problem of perceptual judgment is related to this type of approach. The behavior of the users on the sensor array are transmitted into the decision model finally. Lu et al. [
In the studies described in this section, no time series features have been performed for pattern recognition on pressure arrays. However, time dimension information is worthy of consideration for detection in these scenarios. We propose that the use of three-dimensional CNN in stress map mode detection improve the accuracy. Furthermore the accuracy can also be improved with the consideration of time dimension.
Our goal is to have intelligent control of the mosquito net. The mosquito net is supposed to recognize the intention of the users entering or leaving the mosquito net and, accordingly, lifting the net automatically. We limit the system to standard bed-and-desk beds used in college dormitories in China for simplifying the situation and excluding other possible interference. So in this case, the bed area and the moving range for the users on the bed are both limited. Since the mosquito net still needs to be adjusted manually when entering and exiting the mosquito net regardless of the behavior of the users on the bed, our efforts are focused on avoiding this manual adjustment and realizing intelligent control. Finally, we need to analyze the behavior of the users when they go to bed, and based on their behavior the mosquito net lifts automatically.
The main changes that take place in bed are due to differences in the position of users and pressure distribution on the bed as mentioned earlier [
Firstly, in bed the behavior of the user changes. Secondly, the sensor perceives the changes. Next, the coordinator collects and analyzes the information captured by the sensor and determines the behavior pattern of the user. Features of the information are extracted and fed to the model for classification subsequently. Finally, the mechanical equipment starts the corresponding action according to the behavior recognized for the user.
We use sensor technology, ZigBee wireless networking technology, embedded system development, data communication technology, and mechanical design in the device. The device includes the following design parts.
The design of the data receiving device can judge the behavior of the users based on the received data, both going to bed and getting out of bed included. The data receiving device needs to be able to receive data from various sensors quickly and also have some data analyzed and processed. It also needs to control the working state of the lifting device for the mosquito net. The analysis of data needs to be accurate and fast to control the lifting device for the mosquito net because the process of getting up and going to bed is typically short. In addition, there are diverse behavior patterns for a user in bed beyond his or her sleeping habit. The data receiving device therefore needs to be able to recognize the behavior pattern of the users, and then analyze and judge the behavior of the users. The data receiving device controls the automatic lifting device to raise the mosquito nets when the users go to bed and to allow the mosquito nets fall when the users get out of bed.
The automatic lifting device for the mosquito nets needs to accurately control the rise or fall of the nets according to the data receiving device. Moreover, the automatic lifting device for the mosquito net needs a quick response because the mosquito net takes typically only a short time for the users to go to or get out of bed. Most mosquito nets on the market are very light. Little effort is required to raise these nets. Therefore the typical drive motors on the market can be selected as the power device. In addition, a wide range of choices are available when choosing the configuration of motors. Thus, the lifting of the mosquito net can be refined to match the motion of users getting up or down from the bed.
We do not need to collect data in a large area because the movement of the users on the bed is limited, and the most obvious features are about the changes in the positions of the person on the bed and the pressure on the bed board when he or she moves. Therefore, we need an infrared sensor to collect the data for the changes in position. And also a pressure sensor to collect the data for the changes in pressure is needed.
The sensor data need to be collected in a centralized way for the classification of the intention from the users, which represents a small-scale data acquisition on a large number of sensor nodes. In this context, we selected ZigBee networking to realize data transmission and low-power and high-sensitivity sensors for detection.
The specific models are as follows: (1) The interface conversion chip FT232RL is selected, which can realize the conversion from Universal Serial Bus (USB) to a serial universal asynchronous receiver transmitter (UART) interface, and the conversion of synchronous and asynchronous bit-bang interface modes. (2) The core chip CC2530 is chosen due to its low power consumption and multi-mode adaptation, meanwhile it can be applied in different environments. (3) The infrared sensor hc-sr501is selected with low-power consumption and high sensitivity. (4) The L298N motor drive module is selected. (5) The hx711 is selected as the pressure sensor module of the product.
The ZigBee terminal and the coordinator automatically form a network, and the system is initialized when the power module of the terminal and the coordinator is turned on and powered. First the ZigBee terminal collects data from the infrared sensor and the pressure sensor in turn, and sends the data back to the ZigBee coordinator. Next the data are transmitted through the ZigBee wireless communication module, and the coordinator analyzes the received data and judges the current state of the behavior of the users. The coordinator sends serial data to the host computer for display subsequently.
The pressure on the bed board changes significantly depending on the different behavior of the users on the bed board. These behavior not only comprise sleeping but also turning over and sitting on the bed. Therefore, we must study the placement position and sensitivity threshold of sensors in the equipment. These studies can avoid the influence of operation errors in the judgment of behavior such as the users getting up, getting out of the bed or other unrelated behavior.
We conducted the following experiments and analysis on the placement position and sensitivity threshold of the infrared and pressure sensors in the equipment to improve the accuracy of data acquisition. These experiments and analysis allow us get the best placement of sensors.
The infrared sensor is well suited to judge whether the user goes to bed or gets out of bed as the user typically moves from the end of the bed to the head of the bed when he or she goes to bed. Therefore, the specific detection range of infrared sensors and the sensing effect are tested and analyzed in the experiment. The number of experiments is 20 times per round. The user moves within and outside the detection range. The sensors capture whether the user is within the detection range and collect the relevant data. The analysis of the detection range and effective data rate of the infrared sensor are shown in
Infrared detection range (m) | Effective data efficiency (%) |
---|---|
0.20 | 100 |
0.30 | 100 |
0.40 | 100 |
0.50 | 90 |
0.60 | 75 |
0.70 | 45 |
0.80 | 30 |
The number of experiments is 20 times per round. Valid data is defined as the data captured by the sensor when the user moves within and outside the detection range.
The rate of effective data decreases gradually with the expansion of the infrared detection range as shown in
At bed time the behavior of a user is normally like the way that the head rests on the pillow when he or she goes to bed.
If the detection range is extended to 0.50 m or more, the non-bedtime behavior of the user, consider the act of sitting up or turning on the bed, may also easily trigger the infrared sensor. This behavior makes transmit information from the data receiving device which is consistent with the user leaving the bed. The behavior therefore interferes with the judgment of the data receiving device.
It can be concluded from the graph that the sensitivity of the infrared sensor is very high when the detection range of the infrared sensor is between 0.20 m and 0.40 m. The distance between the head and the side of the bed is precisely 0.05–0.40 m when the user is lying down. Therefore it is reasonable to control the detection range of the infrared sensor to the range 0.20–0.40 m to ensure that the infrared sensor can capture the data of the user accurately.
The size of beds in college dormitories is 190 cm × 90 cm typically. The local pressure under the bed will change as a user gets out of bed.
The typical behavior of the user in bed is not sleep. In addition, the trigger threshold of the pressure sensor has a certain influence on the success rate of judging the out-of-bed behavior of the user. Turning over and changing position have a certain impact on the value detected by the pressure sensor.
Therefore we studied the preset threshold and false trigger rate of the pressure sensor. In this experiment, the pressure sensor is 20 cm away from the bed and 45 cm away from the tail of the bed. The number of experiments was 20 times per round. The experimental data change when setting a different trigger threshold of the pressure sensor.
Trigger threshold of pressure sensor (N) | False triggering rate (%) | Success trigger rate (%) |
---|---|---|
98 | 100 | 100 |
147 | 80 | 100 |
196 | 40 | 100 |
245 | 10 | 100 |
294 | 0 | 100 |
343 | 0 | 100 |
392 | 0 | 80 |
441 | 0 | 55 |
490 | 0 | 0 |
From
The specific placement of the sensor is an additional factor influencing the effective data collected by the pressure sensor. The landing points of knees and legs are concentrated at 20 cm away from the bed because the width of the bed is limited and the students simulate the behavior of getting up and down many times. Therefore, the pressure sensor is placed 20 cm away from the bed in the experiment. The distance between the sensor and the end of the bed will determine whether the user trigger the lifting device by mistake because the pressure changes from the head of the bed to the tail of the bed when the user gets up and down. Therefore, the distance between the pressure sensor and the end of the bed needs further testing.
The following is a series of experiments probing the distance between the pressure sensor and the tail of the bed. In the experiment, the trigger threshold of the pressure sensor is 343N and the number of experiments is set to 20 times per round.
In practical experiments, the successful triggering and false triggering of pressure sensors have different results with the different placement of pressure sensors.
Distance from bed end (cm) | Success trigger rate (%) | False triggering rate (%) |
---|---|---|
25 | 0 | 0 |
30 | 10 | 0 |
35 | 50 | 0 |
40 | 80 | 0 |
45 | 100 | 0 |
50 | 100 | 0 |
55 | 100 | 25 |
60 | 80 | 60 |
65 | 75 | 80 |
70 | 70 | 100 |
75 | 75 | 100 |
It can be concluded that when the pressure sensor is placed at different distances from the end of the bed, it has a significant impact on the successful trigger rate and the false trigger rate (
(1) Feature Extraction
The optical flow is a concept of object motion detection in the computer vision field. It is used to describe the movement of the observation target, surface, or edge relative to the observer. The optical flow method is a method to infer the moving speed and direction of objects by detecting the change of the intensity of image pixels with time. The optical flow can be defined as follows:
where
We use the DIS algorithm [
In the same precision range, DIS is several orders of magnitude faster than the latest methods, which makes it very suitable for real-time applications. So this enables our model to be completed more quickly at the feature extraction stage.
(2) Classifiers
Our approach for the detection of the intention of the users is to use a two-dimensional static method often used by researchers, and we take some of the frames as the input of the classification model in a continuous frame of posture. These methods use many techniques to extract more features for a better classification. However, these efforts ignore the fact that there is a large amount of information between consecutive frames in the time dimension. Here, we use the three-dimensional revolution neural network [
We use a three-dimensional convolution kernel to extract the plane information in consecutive frames and the time dimension information between adjacent frames. We can define a convolution kernel with length, width, and thickness, represent the length, width, and time of the frame plane, respectively. Three-dimensional convolution not only performs convolution operations on the objects in the plane but also on the adjacent frames in the time series. Three-dimensional convolution can sense the change of time sequence by a convolution between two adjacent frames. The thicker the three-dimensional convolution kernel is, the larger the perception field for time will be. The thickness of the 3D convolution kernel must be greater than 1 because the three-dimensional convolution needs to detect the information between adjacent frames.
The convolution kernel first convolutes in three adjacent frames and moves with the time axis subsequently.
The basic structure of the three-dimensional CNN and the essential operation or convolution kernel size of each layer is shown in
The first layer of the network is the input layer, which is supported by ZigBee. The wireless sensor network takes the pressure distribution map of the pressure sensor array as the input and takes 16 consecutive frames as an action sequence which corresponds to a sensor change process with intention tags.
The second layer of the network is the hardwired layer, which uses 16 consecutive frames of the input layer to calculate the optical flow between 15 frames. The method of calculating the optical flow is the DIS algorithm.
The third layer is the convolution layer, which uses two different 5 × 5 × 5 convolution kernels to generate two sets of 46 × 46 frame sequences. The activation function that we choose is Rectified Linear Unit (ReLU).
The fourth and sixth layers of the network are the same as the third layer.
The fifth layer of the network is the lower sampling layer. The max pool method is used for 2 × 2 down-samplings.
The seventh to tenth layers of the network are convolution layers, in which the seventh layer is convoluted by a 3 × 3 × 3 cubic convolution kernel, and the last three layers are continuous 3 × 3 convolutions. The activation method is also ReLU.
In the eleventh layer of the network, the extracted features are used as the input into two fully connected layers which have 240 and 128 neurons each.
The penultimate layer of the network is the dropout layer and this layer is used to prevent overfitting.
The last layer of the network is the output layer, which controls whether the motor is on or off according to the intention of the users and the input pressure distribution diagram.
We use Adam as a gradient descent optimization method. Adam is an update to the RMSProp optimizer. In this optimization algorithm, the continuous average value of the gradient and the second moment of the gradient are used. Given the parameter
where
We use
We set more effective thresholds for the sensors based on the previously described test results, and put these sensors in reasonable positions to build the whole device for our work. The experimental results are described in the following section.
When a user goes to bed, the head and body enter the detection range of the infrared sensor in the process of moving along the end of the bed to the head of the bed. Therefore the infrared sensor can detect the user in the valid range. Next, the ZigBee communication module sends information to the data receiving device. The data receiving device decides that the user is in the bed state when it receives the information. And the data receiving device also knows that there are users in the range of infrared sensor detection. It controls the motor driving module to work and the mosquito nets can drop automatically.
The comprehensive experimental results are shown in
Detection range of infrared sensor (m) | Distance between pressure sensor and bed (m) | Pressure sensor distance from bed end (m) | Trigger threshold of pressure sensor (N) |
---|---|---|---|
0–0.30 | 0.20 | 0.45 | 343 |
The device accurately captures the changes of the behavior of users weighing 52, 56, and 63 kg when they repeatedly use the device to simulate the behavior of going to bed and getting out of bed. The motor drive module controls the rise and fall of mosquito nets and achieves the desired results.
The output data of the UART debugging for the data receiving device are shown in
The infrared sensor transmits data to the data receiving device when the users go to bed and enter the detection range of the infrared sensor. After the analysis, the data receiving device outputs “
In this study, we propose an intelligent mosquito net system using ZigBee technology and a deep learning method. We show that the pressure sensor array and infrared sensor can accurately judge the intention of the users to open or close the mosquito net through the combination of ZigBee and deep learning. We find that, using 3D-CNN to add time dimensional information, the accuracy of pose recognition and classification improves significantly. This shows that using the information of the time dimension based on single frames can increase effectively the learning ability of the deep learning. At the same time, it also means that we can achieve the same level of classification accuracy with fewer sensors compared with previous research. Experiments show that using 3D-CNN can provide strong support for the practical application of our system design. We are also considering using more diversity real-time feature extraction methods to improve the robustness of the system. It can also reduce the reasoning burden of the deep learning model to accelerate the system landing application through using more diversity real-time feature extraction methods.
We thank LetPub (