Vehicle License Plate Recognition System Based on Deep Learning in Natural Scene

: With the popularity of intelligent transportation system, license plate recognition system has been widely used in the management of vehicles in and out of closed communities. But in the natural environment such as video monitoring, the performance and accuracy of recognition are not ideal. In this paper, the improved Alex net convolution neural network is used to remove the false license plate in a large range of suspected license plate areas, and then the projection transformation and Hough transformation are used to correct the inclined license plate, so as to build an efficient license plate recognition system in natural environment. The proposed system has the advantages of removing interference objects in a large area and accurately locating the license plate. The experimental results show that the localization success rate is 98%, and our system is feasible and efficient.


Introduction
With the development of China's transportation in recent years, there are more and more private cars. How to efficiently manage cars has become an urgent problem. At present, the automatic license plate recognition technology is booming. This technology can realize, for example, the management of vehicles entering and leaving the community, the identification of truck license plates on highway crossings, and the collection of parking fees. The license plate recognition in a simple environment has certain practicability.
However, in terms of current research, in the natural environment, there are still many difficulties in automatic license plate recognition technology. For example, bad weathers like haze, rain and snow; As shown in Fig. 1, different lighting conditions; different shooting distances and shooting angles, vehicle speed in the image, and so on. These complex factors have caused problems such as blurred imaging, overexposure, and underexposure in the final captured image. The license plate recognition technology is still not ideal in the natural environment. The key to solving the problem of slow and low accuracy of license plate recognition lies in license plate positioning.
In this paper, in the existing license plate recognition system, the license plate positioning part is optimized, and a new license plate positioning algorithm is used to improve the positioning accuracy, shorten the time and reduce the resource overhead. First, the original image is processed through graying and binarization, and then use the optimized Canny operator to perform the Edge Detection. The optimized Canny operator further improves the accuracy and anti-interference of Edge Detection [1], with fast processing speed and small resource overhead .Combined with morphological processing to extract a large area of suspected license plate area, and after corrosive expansion processing, use Alex Net to remove fake license plates to accurately locate the license plate area; finally set a threshold for cutting and use a template Matching to achieve the segmentation and recognition of license plates. The advantage of this system is that the interference factors in the natural environment are processed accurately, and the license plate is quickly identified after accurately locating the license plate.

Related Work
For the license plate recognition in the natural environment, the key to the solution lies in the positioning of the license plate. At present, there are four main exploration directions for license plate positioning: edge-based, color-based, texture-based, and character-based algorithms [2].
The principle based on edges is to find areas with a larger edge density than the area of the license plate image to be identified, forming edges with different densities. Generally, the edge-based method is to detect rectangular contours because the license plate has a certain aspect ratio [3]. This method is characterized by fast recognition speed and can quickly extract suspected license plate regions, but it is easily affected by the natural environment and has insufficient accuracy. Tan, Jinn-Li combined edge detection with morphology and mathematical operations [4], and performed expansion and corrosion treatment on the screen according to changes in the brightness and area of the license plate to find a rectangle that may be LP. This combination method solves the problem of low recognition accuracy to a certain extent, and has certain feasibility. The edge-based method is characterized by higher detection speed, but this method is more sensitive to unwanted edges and it is not easy to detect complex images.
The color-based method recognizes license plates by locating them in an image. Shi [5] and Zayed et al. [6] proposed a color model classifier. This method uses the color information of the license plate for localization. It has a good recognition rate for photos with a clear license plate color and a simple background, but it is subject to light conditions. Jia et al. used a combination of rectangular features, aspect ratio, and edge density to determine the license plate candidate region, and then used a shift algorithm to segment the color image in the region [7]. This method optimizes in complex scenes Recognition, but the operation is complicated and the recognition speed decreases. Because such methods are sensitive to light and shadow, and are easy to misreport, there is less rese arch now.
The texture-based method is to detect the required area of the pixel distribution in the license plate image. After graying the color license plate image, it scans in rows and columns to count the number of grayscale transitions of the license plate image. By comparison, the positioning of the license plate is finally achieved. A typical method is to use SVM to analyze the color features of the license plate texture [8]. This method converts the image data into vector data and establishes an SVM model for analysis according to specific training parameters. It takes a long time, but the trained model can be fast and accurate. Analysis of license plate textures; Wang, Shen-Zheng uses Adaboost [9] algorithm to build a classifier cascade, uses efficient rectangular features and integral map methods on the underlying feature extraction, extracts Haar-like features, and trains Haar classifiers. This method has extremely high accuracy of license plate recognition. The texture-based method has complicated calculation and poor noise resistance. When in the complex background, or the license plate and the texture characteristics of the body are similar, it is difficult to locate the license plate based on the texture in the image.
The character-based method is to treat the license plate as a character string, while checking for the presence of characters in the image to locate the license plate. For example, scale space analysis is used to extract characters [10]. According to scale invariance and rotation invariance, an image pyramid is established using Gaussian convolution kernels for recognition. It is characterized by excellent recognition speed under appropriate parameters, but it is difficult to find the scale in the image, and it is difficult to choose the appropriate parameters. The character-based method is faster in recognition, but the text in the background of the image will greatly affect the performance of the algorithm.

Natural Scene License Plate Recognition System Based on Deep Learning
As shown in Fig. 2, the system implemented in this paper is divided into four parts, image processing, license plate location, license plate segmentation, and pattern recognition.

Image Preprocessing
As shown in Fig. 3, because in natural scenes it is very difficult to accurately extract license plate boundaries in color original images, the original image should be grayed out, and then contrast enhanced and binarized. The results of Gray processing are shown in Fig. 4. The processed image is detected by the canny operator for edge detection. Compared with other operators, after the canny operator is processed, the edge of the image is very good and it is completely saved.
The principle of the Canny operator [11] is to pass the original image through a Gaussian filter for low-pass filtering. However, Gaussian filtering does not suppress all noise. It is easy to detect impact noise as an edge in a complex environment. Therefore, median filtering is used instead of Gaussian filtering. Then use double thresholds to detect and link edges, and finally form accurate edges. Using median filtering can significantly suppress image noise, better preserve edge information, and achieve smooth effect; when performing gradient calculations, first-order partial derivatives in the 3 × 3 domain are used to find the gradient amplitude and direction, more than the 2 × 2 field, the difference in the horizontal and vertical directions is introduced, which can get more accurate edges and suppress noise interference.

Figure 4: Grayscale image
After several experimental comparisons, as shown in Fig. 5, the edges obtained by optimized edge detection are more complete, and the detailed information can be better saved.

License Plate Positioning
The pre-processed picture still has a lot of noise. In order to enhance the details of the image and the coherence of the image, morphological processing is needed to reduce the noise. For the problem of intermittent and too small positioning areas, expansion and corrosion treatments are required. It receives an image data and a structure. The output value of the pixels in which the background and structure are completely coincident in the image is 1. Finally, the image data after the structure has been etched is returned to achieve the purpose of reducing irrelevant structures.
Most of the corroded image structures are scattered and incoherent. In order to determine the license plate position later, we should further process the images shown in Fig. 5 by a smoothing processing. Here, a closed operation is adopted, that is, the expansion treatment and then the corrosion treatment. The principle of the dilation process is: the foreground image of the binary image is 1, and the background is 0. Assuming that there is a foreground object in the original image, then the process of dilating the original image with a structural element is as follows: traverse each pixel of the original image, then use the center point of the structural element to align the pixel currently being traversed, and then take the maximum value of all pixels in the corresponding area of the original image covered by the current structural element, and replace the current pixel value with this maximum value. Since the maximum value of the binary image is 1, it is replaced with 1, which becomes a white foreground object. For some small breaks in the foreground object, if the structural elements are the same size, these breaks will be connected. The results of corrosion expansion are shown in Fig. 6.  After the initial coarse positioning, the color image to be recognized initially has become a binary image with the license plate as the main structure. However, there are still many suspected license plate areas in the figure. Here, deep learning is used to remove the fake license plate. Experiments prove that the method has high accuracy and simple steps.
Alex Net is a classic model of CNN in the field of image recognition [12]. The specific structure is shown in Fig. 8, the output of the last fully connected layer is the input of a 1000-dimensional Softmax function. Softmax will generate a distribution network of 1000 labels with 8 weighted layers; the first 5 layers are convolution layers, and the remaining 3 layers are fully connected layers. The output of the last fully-connected layer is the input of a 1000-dimensional Softmax, which produces a distribution of 1000 labels. Aiming at the problem that sigmoid gradient saturation leads to slow training convergence, we bring ReLU in Alex Net, which is called here Rectified Linear Units (ReLUs). ReLU is a piecewise linear function. If the value is less than or equal to 0, the output is 0; if it is greater than 0, the output is the same. Using ReLUs in deep learning is much faster than the equivalent Tanh.  According to the needs of the classification, we reduce the output neurons of the classifier from 1000 to 3. In order to prevent the overfitting phenomenon during the training process, we need to process the three RGB channels of the image. Starting from the first layer of the convolution layer, the function is first modified. In the first step, select 96 3D convolution kernels with a step size of 4 and a size of 11 × 11. After operation, get 96 55 × 55 feature plane clusters, and then correct it by activation function.
After correction, we sample from these data. Then, select the Maxpooling operation with a step of 2 and a size of 3 × 3. Then the normalization processing formula is as follows: N is the number of convolution kernels in the convolution layer; I, n, a, β are constant terms. Values can be assigned according to specific experiments. In this paper, i = 2, n = 7, a =10 -4 , and β = 0.75. Experiments show that the training effect after normalization better. Due to the large number of network layers and feature planes, the test results only show part of the features of the third layer network as Fig.10 and Fig. 11.  For the inclination angle problem of the license plate after positioning, using a commonly used license plate correction radon operator to correct the license plate tilt, the precise license plate positioning result can be obtained.

Character Segmentation and Recognition
For the precisely positioned license plate, the binarized license plate part is used to find blocks with continuous text, and if the length is greater than a set threshold, it is cut to complete the character segmentation. There are relatively few words and characters that make up the license plate, and these characters are all printed. Therefore, the template matching algorithm [13] is used to recognize the characters, and a high recognition accuracy can be obtained. The character template library is shown in Fig.12, and the last result of character segmentation is shown in Fig. 13.

Experimental Results and Analysis
Through a large number of experiments to test the improvement scheme of this article, the Alex Net classifier test used 2,810 samples, including 1022 blue license plates, 991 yellow license plates, and 797 fake license plates. Compared with SVM scheme, according to the experimental data of Tab. 1 and Tab. 2, the accurate positioning rate of the final license plate reached 98.3%, and the wrong positioning rate of final license plate reduced to 2.58%. It is proved that this algorithm has good recognition performance and the system is feasible and effective.  Because Canny Edge Detection combined with morphological processing is used for preliminary positioning, the time taken by the Alex Net classifier to scan the full image is greatly reduced. Because the license plate recognition system in this paper is based on two different features, it can be executed simultaneously under a multi-core CPU without affecting each other, reducing time and increasing the operating speed. This method solves to a certain extent the shortcomings of the current license plate positioning technology in a complex environment, such as low positioning accuracy and long time, and enables the license plate positioning technology to achieve higher positioning in natural scenes with many vehicles, many people, and large driving angle Accuracy. At the same time, with the continuous improvement of CPU performance, the calculation cost of the Alex Net classifier will also decrease, so the system has a large room for improvement in engineering applications.

Conclusion
Incorporating deep learning theory and Alex Net model, into the license plate location system, not only improves the accuracy of license plate location, but also reduces the time for full image scan of Alex Net classifiers when using Canny Edge Detection method combined with morphological processing to perform preliminary positioning. Combined with a large number of experimental results, it is proved that the system has a high positioning success rate, use less time and fewer resources, and has certain practicability.

Conflicts of Interest:
The authors declare that they have no conflicts of interest to report regarding the present study.