Licence Plate Detection Using Machine Learning

License Plate Recognition (LPR) is one of the tough tasks in the field of computer vision. Although it has been around for quite a while, there still lies the challenges when we have to deal with; the harsh environmental conditions like snowy, rainfall, windy, low light conditions etc. as well as the condition of the plates which includes the bent, rotated, broken plates. The performance of the recognition and detection frameworks take a significant hit when it is concerned with these conditional effects on the license plate. In this paper, we introduced a model to improve our accuracy based on the Chinese Car Parking Dataset (CCPD) using 2 separate convolutional neural networks. The first CNN will be able to detect the bounding boxes for the license plate detection using Non-Maximus Suppression (NMS) to find the most probable bounding area whereas the second one will take these bounding boxes and use the spatial attenuation network and character recognition model to successfully recognize the license plate. First, we train the CNN to detect the license plates, then use the second CNN to recognize the characters. The overall recognition accuracy was found to be 89% in the CCPD dataset


Introduction
Licence Plate Detection and Recognition (LPR) is a crucial element implied in the fields of vehicular transportation systems and is a difficult and necessary.It is implemented in traffic management systems, digital security systems, police work, vehicle recognition, parking management, etc.Most of the methods in the LPR use the following methods of implementation; analyse the image or videos captured by the capturing device, process the images using digital image processing, recognise the pattern using pattern recognition or alternative technologies to get the vehicle plate registration number and colour information of the plate.In the real world, the images are taken over days and nights in extremely complicated atmosphere conditions with variable illuminations and completely different lighting conditions.The key process includes localization, segmentation and finally the recognition of the plate which becomes a difficult task to perform.In this paper, we introduce a method that works using the convolutional neural network for the detection of plate region and finally the recognition of characters from the licence plate.This paper presents the following contributions to the pre-existing works: 1) We introduce a real-time convolutional network-based method to detect the licence plate by training it with a huge number of images of different weather conditions and other various defects.
2) We use the spatial networks to make the recognition performance higher.
The rest sections of the paper is described as follows.Section II, gives a brief about the literature survey conducted.Section III explains the methodology and the approach used.Section IV is about the proposed system, dataset description, data analysis, data pre-processing, data exploration.Section V is about result analysis followed by conclusion in Section VI and References.

Related Works
Chris Henry and Sang-Woong lee [1] used deep networks in multinational license plate classification and detection.Unified character recognition was the main method that was applied in recognizing the nline at: le o b ila Ava -1091 -characters using the YOLOv3-SPP.The features are combined to create a spatial pyramid of features.These features will be used in the recognizing the character features from the image.The plate regions are also searched using the YOLO framework itself.Xie and Jen [2] used a multi directional approach on top of the YOLO based framework.This combined framework formed a system to detect the plate.CNN was used as this provided a high accuracy in the detection and the detection time being small allowed it to be used in the real time scenarios.They used rotational angle prediction and Intesection Over Union (IoU) for the detection.This significantly reduced the problems that were faced with the rotated plates.L. Xie [3] introduced a new method to validate the intersection over union by the use of penalty factors.To get the precision higher in detection they used the intersection over union using 2 rotational angles and this was used in the penalty factor.They also used a CNN to find the region for the attention.Yongjje Zou, Youngjun Zhang, Jun Yan, Xiaoxu Jiang [4] used the Bi-LSTM for a robust license plate recognition model.It had significant advantages like activating the regional features of characters and extracting the character of license plates, then using the Bi-LSTM to locate each license plate character and also their location information.1D-Attention mechanism was the key to extract only the useful character.This reduced useless features to increase the effective recognition precision of character features from license plates.
V. Pustokhina [5] used the 3-step process to detect the license plate.First the license plate is localized and then Bersen Algorithm and CCA models are used for the licence plate detection.The characters are then segmented using clustering and Krill Herd Algorithm along with optical K-Means.The CNN Model is then used for the recognition of characters in the image.C. Y. Suen [6] used separate YOLO frameworks for the detection and recognition process.The system used separate CNN for the detection of the licence plate and then separate one for the character recognition process.Various data augmentation techniques were applied to increase the dataset size for training the model to get a better accuracy.
Ching Yee Suen [4] combined the bi-directional LSTM and the 1D attention using heat maps and the position to detect the licence plates.It contains the region extraction and the useful and less useful region classification from the image.The features are taken from the useful region and it is used to detect the characters.V. Pustokhina [5] used the 3-step process to detect the license plate.Bersen Algorithm is used for localization of the licence plate and CCA models are used for detection.The characters are then segmented using clustering and Krill Herd Algorithm along with optical K-Means.The CNN Model is then used for the recognition of characters in the image.approach In this section, architecture of the 2 CNN's that we have used for detection and the underlying recognition network for the license plate is shown.

Detection Structure
Neural Networks have proven to be efficient in finding the features.Deep Neural Networks even take this to a much further domain.Real world applications require a high precision as well as faster processing times.In today's world we have seen a great improvement in the precision as well execution times due to the advancement of the computer vision.Li et al. [16] used a cascaded CNN method which was able to maintain a high accuracy rate but with the lower time consumption by cascading the lower performance classifier to create a strong classifier.Our system is based on cascaded CNN method [17] and it is implemented using the MTCNN algorithm [18].The MTCNN are generally used for face detection and recognition.We have modified the algorithm so that it can be used in license plate detection.The detection network consists of 2 CNN's.The first CNN will be used to find the bounding boxes for the license plates.These boxes are nothing but the coordinates for the license plate.The input images are scaled to various sizes and they will generate a feature pyramid.These give us candidates of the licence plate and the regression vector.All these candidates are then combined using the non-Maximus suppression.NMS [18] is a technique where use the regression method to find the most optimal features for our detection purpose.Each feature has a score which will give a measure of its precision.For the choosing of the windows the one with the highest score is considered and other windows which are overlapped in the image is discarded.

The modified architecture of P-Net and O-Net can be shown below
Available online at: https://jazindia.com-1092 -

Training The Network
Here we employ 3 simultaneous tasks for training our CNN.The tasks include plate classification, bounding box regression and plate localization.The tasks will generate various bounding boxes from the smaller CNN and then refined again using the second CNN and merged using non-Maximus suppression.For the training purpose, we take the images from the dataset containing all the different types of challenges.The images consist of rotating plates, broken plates, low illumination plates and other various challenging images which are harder for the detection algorithms.This improves the performance of the model.

Recognition Structure
The recognition network consists of a small basic block that is integrated into the CNN.The architectures of the small basic block and the underlying recognition network are shown below

B. EVALUATION METRICS
The dataset CCPD which we have been using contains only the license plate location embedded in the label of the image.Hence, we can get compare our predicted location with the original location for the license plate which is in the annotation.The final detection model will use all the previously explained techniques to give a final output location.The only way to measure whether our output is correct or not is if it matches the annotated name, then it is considered as correct; Else it considered as incorrect.The annotated number of license plate can be taken as the ground truth or the actual value.We will consider how many our model has predicted correctly.Also, one thing to consider is that some of the images don't contain any plate number in them.Hence incorrectly annotated.We will not consider them in the ground truth values.The regression using the IoU values will give us a score in percentage determining their accuracy.If the accuracy is 70%, we will consider them to be correct.The Accuracy precision can be defined by the below-given formula The final recognition accuracy can be calculated from the above given formula.Since the model will predict something from the image, it will be matched with the characters after decoding from the label.

Results and Discussion
Here, we will compare the accuracy of our result set with the existing methods that have already been applied to the CCPD Dataset.The system was implemented in PyTorch; a tool to implement our neural network using visual studio code and google collab notebook.The training and validation purposes were carried out in the google collab notebook.For the detection framework, we have chosen 15 epochs.This can be shown as below   As we can see that the execution time for each image is nearly 0.15s and max of 0.35s.the graph shows execution time taken by each image when the system was implemented over 1k images.The final model accuracy was tested over the various subsets of images containing all the various conditions of the license plate as well as the challenging ones.Each of the algorithms have been already tested and a base comparison has been provided in the CCPD dataset.We compare our results with the existing ones.They have been shown in the below table.

Conclusion
License plate detection and recognition face the challenge of getting higher accuracy with lower execution times to be able to be used in the real world.Smaller models can help the system to be sustainable in a long time.Challenges have always emerged to improve the accuracy within reasonable execution times and the main goal of this paper also extends to the creation of a system that could be used in real-time.On using the model for the detection and recognition, the overall

Fig 2 :
Fig 2: Modified O-Net ArchitectureHere convolution layer is using stride of 1 and the pooling layer uses a stride of 2.

Fig 4 .
Fig 4. Per image execution time

Table 1 :
Recognition Network Backbone

Table 2 :
[19]l Basic BlockWe mainly use and compare the performance of the Chinese Car Parking Dataset (CCPD)[19]which have been already applied to this dataset.The CCPD dataset contains nearly about 250K images with different conditions like different backgrounds, different rotation angles of the image, different shooting angles of the license plate and various other plates which were taken in different environmental conditions.It also contains the sub folders for various types of images.The folders include base, DB, FN, Rotate, Weather and Challenge.Each of these contains the various types of images for our purpose.