Python-Powered Number Plate Recognition: Unleashing the Potential of Azure Custom Vision and Azure Computer Vision
Tags: azure, cognetiveservices, python
Published: 2023-06-14 19:45:38

**Introduction**

Number Plate Recognition (NPR) has become increasingly significant in numerous domains, including law enforcement, traffic management, and parking systems. By automating the process of extracting and identifying license plate numbers, NPR systems streamline operations, enhance security, and enable efficient data analysis. In this blog post, we will explore a comprehensive approach to Number Plate Recognition that combines the power of Azure Custom Vision and Azure Computer Vision technologies.

**Approach**

Our approach to Number Plate Recognition involves two fundamental components: Azure Custom Vision and Azure Computer Vision. Azure Custom Vision, a cloud-based image recognition service, enables us to create a robust and accurate license plate detection model. Conversely, Azure Computer Vision, powered by Azure Cognitive Services, empowers us to extract the plate number from the detected license plate regions.

**1. Training the License Plate Detection Model using Azure Custom Vision**

The first step in our approach is to train a license plate detection model using Azure Custom Vision. This process entails collecting a diverse dataset of labeled license plate images, covering various license plate sizes, styles, lighting conditions, and angles. With Azure Custom Vision, we can easily upload the dataset, train a machine learning model, and fine-tune it to accurately detect license plates within images.

From the Azure Portal, search _Cognetive Services multi service account_ and create an account.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x44j06z5yninz9hhwqb8.png)

For this example, the instance name is CarPlateDetection.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/awzuxhoogvy1g8uvaayn.png)

Open another tab and go to the [Custom Vision Portal](https://www.customvision.ai/). You must sign in with the Microsoft account that used earlier. Click on _New Project_.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0z0httqokw5mcvdrpj6d.png)

Type the project name, description and select the resource. Click on _Object Detection_ as Project Type and _General [A1]_ as Domain, then Create Project.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uouw7kjkgx2y28u63jz0.png)

Click on _Add images_ to upload the car images where the plate number is visible. Search at least 20 images for this example.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zxlb2g1p5ssy5qo0kudb.png)

You will see the images you uploaded.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qvcul5bbnqyya9tal3g3.png)

Now, click on each one to select the region of the plate number you want the Machine Learning model detects. You must type the tag name after you select the region.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f0fbdatpt2sd4xb97pz5.png)

After you finished to tag all the images, click on _Train_.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bcgnxg5nhk3q79wqqqvl.png)

Chosse _Quick Training_ an click on _Train_.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dwc0vt5and2zv4375fsv.png)

This can take between 10 and 15 minutes based on the number of images you tagged.

You will see three metrics:

- **Precision:** It's the percentage of identified classifications that were correct. For example, if the model identified 100 images as dogs and 99 of them were actually dogs, then the precision is 99 percent
- **Recall:** It's the percentage of actual classifications that were correctly identified. For example, if there were actually 100 images of apples, and the model identified 80 as apples, the recall is 80 percent.
- **Mean average precision (mAP):** It's the average value of the average precision (AP). AP is the area under the precision/recall curve (precision plotted against recall for each prediction made).

The _Probability Threshold_ is the desired level of confidence that a prediction needs to have in order to be considered correct. When you interpret prediction calls with a high probability threshold, they tend to return results with high precision at the expense of recall. That is, the detected classifications are correct, but many remain undetected. A low probability threshold does the opposite: most of the actual classifications are detected, but there are more false positives within that set. With this in mind, you should set the probability threshold according to the specific needs of your project. By default, the probability threshold is 50% and can be set between 0% and 100%.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4cgawjf6gxbvz6wejwmm.png)

You can test the model by clicking on _Quick Test_. Upload any car image to test the model.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wk4p1lmalmb39fu764tg.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w4py2ooj2cw4hn5th7vk.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/udcfq903fx2xefvu3nf5.png)

Once you are comfortable with the testing results, click on _Publish_ and type the model name, then select the prediction resource. Finally, click on _Publish_.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j5ytpzli8phc12qbfkne.png)

Click on _Prediction URL_ to see the URL and Prediction Key.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1dvf2g4e0ljxjco6n4qa.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ajc455vxygdhz7eatkqx.png)


**2. Preprocessing and Localization**

After training the license plate detection model, we need to preprocess the input images to enhance their quality and reduce noise. Azure Cognitive Services provide powerful image preprocessing capabilities, such as image resizing, contrast adjustment, and noise reduction, which can be applied to optimize the image for subsequent processing. Once the preprocessing is complete, the license plate detection model is utilized to locate license plate regions within the images.

We'll create a Python Notebook to call the Azure Custom Vision and Azure Computer Vision APIs.

```python
import requests
import cv2
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from PIL import Image
from numpy import asarray
```
Take a look to [Custom Vision Prediciton 3.0](https://westus2.dev.cognitive.microsoft.com/docs/services/Custom_Vision_Prediction_3.0/operations/5c82db60bf6a2b11a8247c19) for further details.

```python
custom_vision_imgurl = 'https://<resource_name>.cognitiveservices.azure.com/customvision/v3.0/Prediction/<project_id>/detect/iterations/<published_name>/image'
api_key = '<YOUR_PREDICTION_KEY>'
```

```python
data = open('Cars326.png', 'rb').read()
# decode the image file as a cv2 image, useful for later to display results
img = cv2.imdecode(np.array(bytearray(data), dtype='uint8'), cv2.IMREAD_COLOR)
```

```python
custom_vision_headers = {
    'Content-Type': 'application/octet-stream', 
    'Prediction-Key': api_key
    }
    
custom_vision_resp = requests.post(url = custom_vision_imgurl, 
    data = data, 
    headers = custom_vision_headers).json()
```

```python
# inspect the top result, based on probability 
hit = pd.DataFrame(custom_vision_resp['predictions']).sort_values(by='probability',ascending=False).head(1).to_dict()

print(hit)
```

```json
{
   "probability":{
      "0":0.99115235
   },
   "tagId":{
      "0":"c5b7e0da-7c90-4e63-bf6f-b299f1c6492a"
   },
   "tagName":{
      "0":"Plate"
   },
   "boundingBox":{
      "0":{
         "left":0.3620827,
         "top":0.6698199,
         "width":0.2792757,
         "height":0.08204675
      }
   }
}
```

```python
# extract the bounding box for the detected number plate 
boundingbox = list(hit['boundingBox'].values())[0]
l, t, w, h = (boundingbox['left'], 
    boundingbox['top'], 
    boundingbox['width'], 
    boundingbox['height'])

# extract bounding box coordinates and dimensions are scaled using image dimensions 
polylines1 = np.multiply([[l,t],[l+w,t],[l+w,t+h],[l,t+h]], 
    [img.shape[1],img.shape[0]])

# draw polylines based on bounding box results
temp_img = cv2.polylines(img, np.int32([polylines1]), 
    isClosed = True, color = (255, 255, 0), thickness = 2)

# display the original image with the plate region
plt.imshow(cv2.cvtColor(temp_img, cv2.COLOR_BGR2RGB))
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bu915irhmv55gorzsrxt.png)

```python
# crop the image to the bounding box of the plate region
crop_x = polylines1[:,0].astype('uint16')
crop_y = polylines1[:,1].astype('uint16')

img_crop = img[np.min(crop_y):np.max(crop_y), 
    np.min(crop_x):np.max(crop_x)]

# display the detected plate region
plt.imshow(cv2.cvtColor(img_crop, cv2.COLOR_BGR2RGB))
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2fp05ek61emgrn5vuf3x.png)

**3. License Plate Extraction using Azure Computer Vision**

Once the license plate regions are identified, we employ Azure Computer Vision techniques to extract the actual plate numbers. This step involves applying image segmentation algorithms to isolate the characters and digits on the license plates. With Azure Computer Vision's OCR capabilities, we can accurately extract the alphanumeric information from the segmented regions.

Before calling the Azure Computer Vision API, we must ensure the image we send has at least 50px of height.

```python
img_crop_height = img_crop.shape[0]
if img_crop_height < 50:
      pil_image = Image.fromarray(img_crop)
      img_crop_width = img_crop.shape[1]
      difference = 50 / img_crop_height
      resized_dimensions = (int(img_crop_width * difference), int(img_crop_height * difference))
      pil_image_resized = pil_image.resize(resized_dimensions)
      img_crop_resized = asarray(pil_image_resized)

      plt.imshow(cv2.cvtColor(img_crop_resized, cv2.COLOR_BGR2RGB))
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j82v9a1zwnnxpblypws2.png)

```python
computer_vision_imgurl = 'https://<resource_name>.cognitiveservices.azure.com/computervision/imageanalysis:analyze?api-version=2023-02-01-preview&features=read'
```

```python
crop_bytes = bytes(cv2.imencode('.png', img_crop_resized)[1])
 
# make a call to the computer_vision_imgurl
computer_vision_resp = requests.post(
    url=computer_vision_imgurl, 
    data=crop_bytes, 
    headers={
        'Ocp-Apim-Subscription-Key': api_key, 
        'Content-Type': 'application/octet-stream'}).json()
```

```json
{
   "readResult":{
      "stringIndexType":"TextElements",
      "content":"DL8CX 48.50",
      "pages":[
         {
            "height":50.0,
            "width":224.0,
            "angle":0.5536,
            "pageNumber":1,
            "words":[
               {
                  "content":"DL8CX",
                  "boundingBox":[
                     5.0,
                     12.0,
                     103.0,
                     13.0,
                     102.0,
                     43.0,
                     4.0,
                     42.0
                  ],
                  "confidence":0.677,
                  "span":{
                     "offset":0,
                     "length":5
                  }
               },
               {
                  "content":"48.50",
                  "boundingBox":[
                     127.0,
                     14.0,
                     208.0,
                     14.0,
                     207.0,
                     44.0,
                     127.0,
                     43.0
                  ],
                  "confidence":0.668,
                  "span":{
                     "offset":6,
                     "length":5
                  }
               }
            ],
            "spans":[
               {
                  "offset":0,
                  "length":11
               }
            ],
            "lines":[
               {
                  "content":"DL8CX 48.50",
                  "boundingBox":[
                     4.0,
                     11.0,
                     211.0,
                     13.0,
                     211.0,
                     43.0,
                     4.0,
                     41.0
                  ],
                  "spans":[
                     {
                        "offset":0,
                        "length":11
                     }
                  ]
               }
            ]
         }
      ],
      "styles":[
         
      ],
      "modelVersion":"2022-04-30"
   },
   "modelVersion":"2023-02-01-preview",
   "metadata":{
      "width":224,
      "height":50
   }
}
```

In the above response, we can see that the plate number is _DL8CX 48.50_ which is present in readResult.content.

You can find the source code [here](https://gist.github.com/cristofima/600907027d5f58e017ba07bdfa080b47).

**4. Post-processing and Analysis**

After successfully extracting the license plate numbers, additional post-processing techniques can be applied to refine the results. These may include noise removal, character recognition validation, and formatting adjustments to ensure the extracted plate numbers are accurate and formatted correctly. Once the data is ready, it can be further analyzed, stored, or used for various purposes, such as database integration or real-time monitoring systems.


**Conclusion**

Number Plate Recognition using Azure Custom Vision and Azure Computer Vision technologies offers a comprehensive solution for accurate license plate detection and extraction. By leveraging the capabilities of Azure's cloud-based services, we can automate the process of recognizing and extracting license plate numbers, opening up a wide range of possibilities for applications in traffic management, parking systems, and law enforcement. In the upcoming sections of this blog series, we will delve deeper into each step of the approach, exploring the underlying techniques and demonstrating their implementation using Azure Custom Vision and Azure Computer Vision. Stay tuned to unlock the potential of Number Plate Recognition and revolutionize your image processing workflows with Azure's advanced AI capabilities.

**Thanks for reading**

Thank you very much for reading, I hope you found this article interesting and may be useful in the future. If you have any questions or ideas that you need to discuss, it will be a pleasure to be able to collaborate and exchange knowledge together.