Synthetic dataset generation for machine learning by Blender: my first trial
Tags: blender, tensorflow, machinelearning
Published: 2022-04-11 15:18:10

TL;DR This is notes of my first trial of synthetic dataset generation for machine learnings. Just generating simple torus rendered images and trained machine learning model to detect them.

# Training method
I referred [this article about training a model for Tensorflow.js](https://blog.tensorflow.org/2021/01/custom-object-detection-in-browser.html) to make a model. I tested with the kangaroo dataset and tomato dataset on Kaggle. Both worked fine and the trained model gave me good predictions. I applied the procedure to my dataset generated by Blender.

# Steps
## Set up scene
Let's put a torus at the origin of the scene. I left the default point light and the camera as it was.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d22trgmtxwlpobjmerki.png)

Next, go to the `Shading` tab and create a material. By default, `Principled BSDF` is used as the surface shader. I left as it is. As I talk in the next section, I changed the `Base Color` randomly in the `Principled BSDF` to set the color of the torus.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ye479zum2rddq90x1sz.png)

## Scripting
On Blender, we can write python scripts to control objects' properties, set up rendering parameters etc. I automated rendering images, moving / rotating the torus and changing colors randomly. After setting those parameters, I calculated the bounding boxes (x min, y min, x max, y max) of torus in each rendered images to generate annotation dataset for training and rendered images as inputs. I generated 600 images for training and 200 images for testring. Let me share a part of my script. (The full script is on [Github](https://github.com/ku6ryo/synthetic-dataset-torus/blob/master/blender/script.py). Thanks for [a Blender forum article](https://blender.stackexchange.com/questions/7198/save-the-2d-bounding-box-of-an-object-in-rendered-image-to-a-text-file). I use the code on the article to calculate the bonding box.)
```python
base_dir = "d:/rendering_result"
geometry_file_path = os.path.join(base_dir, "geometry.txt")
donut = bpy.data.objects["Torus"]
with open(geometry_file_path, "w") as file:
    for i in range(800):
        # Setting rotation
        donut.rotation_euler[0] = random() * math.pi
        donut.rotation_euler[1] = random() * math.pi
        donut.rotation_euler[2] = random() * math.pi
        # Setting rotation
        donut.location.x = random()
        donut.location.y = random()
        donut.location.z = random()
        # Setting material color
        color = (random(), random(), random(), 1)
        donut.active_material.node_tree.nodes['Principled BSDF'].inputs["Base Color"].default_value = color
        
        # Naming a file and render an image
        f = "image" + str(i) + ".png"
        path = os.path.join(base_dir, f)
        bpy.context.scene.render.filepath = path
        bpy.ops.render.render(write_still = True)
        
        # Calculating the bounding box
        b = camera_view_bounds_2d(bpy.context.scene, bpy.context.scene.camera, bpy.data.objects['Torus'])
        
        # Generating annotation CSV row and writing it in a file
        row = "%s,%i,%i,donut,%i,%i,%i,%i" % (f, 512, 512, b.x, b.y, b.x + b.width, b.y + b.height)
        file.write(row + "\n")
```

The followings are the example images and the annotation data
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zifdysmp5dbxs5ubch99.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2skj2pk9su36cqoxjguq.png)

```
filename,width,height,class,xmin,ymin,xmax,ymax
image0.png,512,512,donut,207,143,333,285
image1.png,512,512,donut,262,157,324,311
image2.png,512,512,donut,253,122,411,267
image3.png,512,512,donut,268,146,420,301
image4.png,512,512,donut,243,185,401,286
...
```

## Training with the dataset
I do not explain the details of my training because the steps are written in [the article I referred](https://blog.tensorflow.org/2021/01/custom-object-detection-in-browser.html). However, I trained my models on my PC although the article is using Colab. [My notes](https://github.com/ku6ryo/tomatos) may help if you want to make your models in your local PC.

I used `1000` as the training step. First I tried 7500 as the same as the kangaroo model training in the article, but it was too much for this training.

# Result
With the 200 testing dataset, I checked detection results. Almost all torus are detected with confidence over 70% confidence.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/as1izmmpdjgbqnzavtvz.gif)

However, the model did not recognized the torus in the images taken by my webcam (I took pictures of my PC screen with the webcam). I guess the training torus in the dataset is too clean comparing to the real world. I should include real photo image as the background and some noise, motion blur and so on in the dataset.

Through this process, I learned synthetic dataset generation somewhat work but I still have many improvements I should introduce to use my model in the real world (like AR). Please share your experiences about synthetic dataset generation as comments!