Detecting Welding Defects in Steel Plates using Computer Vision Algorithms
Tags: machinelearning, deeplearning, computervision, tutorial
Published: 2021-09-11 14:05:17

---
description : Detecting and measuring severity of welding defects using CNNs and UNets
---

## Table of Contents

1. Introduction
2. Prerequisites
3. Image Segmentation
4. Image Moments
5. Understanding the Data
6. Method and Algorithm Used
7. Results
8. Conclusion
9. Co-Authors
10. References

## 1. Introduction

Welding defects can be defined as weld surface irregularities, discontinuities, imperfections, or inconsistencies that occur in welded parts. Defects in weld joints could result in the rejection of parts and assemblies, costly repairs, significant reduction of performance under working conditions and, in extreme cases, catastrophic failures with loss of property and life.

Furthermore, there are always certain flaws in the welding due to the inherent weakness in welding technology and the characteristics of metals.

It is important to evaluate the weld quality, as welded joints are often the locations of crack initiation due to inherent metallurgical geometrical defects, as well as heterogeneity in mechanical properties and the presence of residual stresses.

In practice, it is practically impossible to obtain a perfect weld and, in most circumstances, it is not necessary to provide the adequate service functions required. However early detection and segregation is always preferred to mishaps.

Using our algorithms, we can easily detect faults in welding by the images and also measure the severity of each fault precisely. This will further help in faster image recognition and avoid adverse situations from arising.

It was found, using Convolutional Neural Networks algorithm and U-Net architecture making the process much more efficient. Resulting in an accuracy of 98.3% by the end of the work.

## 2. Prerequisites

- Basic understanding of Machine Learning
- Basic idea of Convolutional Neural Networks
- Understanding of Convolution, Max Pooling and Up Sampling operations
- Idea of U-Net architecture
- Basic understanding of skip connections as in residual blocks (Optional)
- Working knowledge of ConvNets with Python, TensorFlow and Keras library (Optional)

## 3. Image Segmentation

Segmentation partitions an image into distinct regions containing pixels with similar attributes. To be meaningful and useful for image analysis and interpretation, the regions should strongly relate to depicted objects or features of interest.

The success of image analysis depends on reliability of segmentation, but an accurate partitioning of an image is generally a very challenging problem.

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t16vlpbdie9nynuriv8k.png'>
<figcaption>A chest x-ray with the heart (red), lungs (green), and clavicles (blue) are segmented</figcaption>

## 4. Image Moments

An image moment is a certain particular weighted average of the image pixels’ intensities. Image moments are useful to describe objects after segmentation.

Simple properties of the image which are found via image moments include:

- Area (or total intensity)
- Centroid
- Information about its orientation.

## 5. Understanding the Data

The [dataset](https://github.com/malakar-soham/cnn-in-welding/tree/master/Code/dataset) contains two directories. The raw images are stored in `images` directory and the segmented images are stored in `labels` directory.

Let’s visualize the data:

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j8fc0tq5y2zpyzzzft8t.png'>
<figcaption>Raw Image from images</figcaption>

These raw images above are RGB images which have to be used for training the model as well as testing the model. These pictures vary in dimensions. Intuitively, the darker portions are welding defects. The model needs to perform Image Segmentation on these images.

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2bnoncpb9kou2n3kzffr.png'>
<figcaption>Binary Image from labels</figcaption>

These images from `labels` directory are binary images or ground truth labels. This is what our model must predict for the given raw image. In binary images the pixels have either a `high` value or a `low` value. The white region or the `high` values denote the defective regions and the black region or the `low` values denote no defect.

## 6. Method and Algorithm Used

The architecture we are going to use for this problem is U-Net. We are going to detect faults and measure the severity of these welding images in three broad steps:

- Image Segmentation
- Showing Severity using Colour
- Measuring Severity using Image Moments

### Training the Model

The following is the U-Net architecture we used for our model:

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d5hu89gtl0vc1aefnrj0.png'>
<figcaption>The U-Net Architecture used</figcaption>

### Points to Note:

- Each blue box corresponds to a multi-channel feature map
- The number of channels is denoted on the top of the box.
- The (x,y) dimensions are provided at the lower left edge of the box.
- The arrows denote different operations.
- The name of the layer is provided below the layer.
- C1, C2, …. C7 are the output layers after Convolutional operation
- P1, P2, P3 are the output layers of Max Pooling operation
- U1, U2, U3 are the output layers of up-sampling operation
- A1, A2, A3 are the skip connections.
- The left-hand side is the contraction path, where regular convolutions and max pooling operations are applied
- The size of the image gradually reduces while the depth gradually increases.
- The right-hand side is the expansion path, where (Up Sampling) transposed convolutions along with regular convolutions operations is applied
- In the expansion path, the size of the image gradually increases and the depth gradually decreases
- To get better precise locations, at every step of the expansion we use skip connections by concatenating the output of the transposed convolution layers with the feature maps from the Encoder at the same level:
A1 = U1 + C3
A2 = U2 + C2
A3 = U3 + C1
- After every concatenation we again apply regular convolutions so that the model can learn to assemble a more precise output.

{% gist https://gist.github.com/malakar-soham/5a8979d850c0e63c30c1788c6d2966b7 %}

Model is compiled with Adam optimizer and we use binary cross entropy loss function since there are only two classes (defects and no defects).

We use a batch size of 10 with 100 epochs(The number of times the model is run on all inputs).

Please note that there could be a lot of scope to tune these hyper-parameters and further improve the model performance.

### Testing the model

Since the model takes input of dimension 512x512x3, we have resized the input to that dimension. Next, we have normalized the image by dividing it by 255 for faster computation.

The image has been fed to the model for predicting the binary output. In order to amplify the intensities of the pixels the binary output has been multiplied by 1000.

The image is then converted to 16-bit integer for easy image manipulations. After which an algorithm detects the defects and visually marks the severity of the defects through color grading along with assigning weights to the pixel with defects depending on their severity. The Image Moment is then calculated on this image considering the weighted pixels.

The image is finally converted back to 8-bit integer and the output image is displayed with color grading and its severity value.

{% gist https://gist.github.com/malakar-soham/474d444545df86d850c956b7d29cbf76 %}

## 7. Results

The visual metric we are using for severity detection is colours

In the image, the colours:

1. Green signifies areas with severe defects.
2. Blue corresponds to areas with more severe defects.
3. Red areas show the most severe defects.

The 0th moment is displayed as a percentage alongside the output image as an empirical measure of severity.

The following are three random samples which shows the raw input, the ground truth and the output generated by our model.

### Sample 1:

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o9a1vhykbq2fy94zlvrn.png'>
<figcaption>Raw Image</figcaption>

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2fk1c9yighhgulu9s9da.png'>
<figcaption>Binary Image (Ground Truth)</figcaption>

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dscleg607y04ytz1hsjx.png'>
<figcaption>Predicted Output with Severity</figcaption>

### Sample 2:

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xi813xybk7hoda2ax40k.png'>
<figcaption>Raw Image</figcaption>

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5eo6gry0ujoyk9gkt9nv.png'>
<figcaption>Binary Image (Ground Truth)</figcaption>

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cu7ws3kz6eeh3t3a80yu.png'>
<figcaption>Predicted Output with Severity</figcaption>

### Sample 2:

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j8fc0tq5y2zpyzzzft8t.png'>
<figcaption>Raw Image</figcaption>

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2bnoncpb9kou2n3kzffr.png'>
<figcaption>Binary Image (Ground Truth)</figcaption>

<img src='https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k511x2cuidrr0ebbkbns.png'>
<figcaption>Predicted Output with Severity</figcaption>

## 8. Conclusion

Thank you for interest in the article. Please leave your valuable feedback and suggestions 😃

Full code on my GitHub repository [here](https://github.com/malakar-soham/cnn-in-welding).

## 9. Co-Authors

Soham Malakar — Connect with me on LinkedIn, [here](https://www.linkedin.com/in/malakar-soham/)

Swastik Malakar — Connect with me on LinkedIn, [here](https://www.linkedin.com/in/malakar-swastik/)

## 10. References

https://www.cs.auckland.ac.nz/courses/compsci773s1c/lectures/ImageProcessing-html/topic3.htm#adaptive

https://medium.com/r/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FImage_moment

https://medium.com/r/?url=https%3A%2F%2Ftowardsdatascience.com%2Funderstanding-semantic-segmentation-with-unet-6be4f42d4b47

https://www.sciencedirect.com/topics/materials-science/welding-defect

https://domingomery.ing.puc.cl/material/gdxray/