Building Natural Flower Classifier using Amazon Rekognition Custom Labels The Complete Guide with AWS Best Practices
Tags: aws, machinelearning, computervision
Published: 2020-11-17 07:11:13

# **Introduction**
Building your own computer vision model from scratch can be fun and fulfilling. You get to decide your preferred choice of machine learning framework and platform for training and deployment, design your data pipeline and neural network architecture, write custom training and inference scripts, and fine-tune your model algorithm’s hyperparameters to get the optimal model performance.

On the other hand, this can also be a daunting task for someone who has no or little computer vision and machine learning expertise. This post shows a step-by-step guide on how to build a natural flower classifier using [**Amazon Rekognition Custom Labels**](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/what-is.html) with AWS best practices.

# **Amazon Rekognition Custom Labels Overview**
**Amazon Rekognition Custom Labels** is a feature of [Amazon Rekognition](https://aws.amazon.com/rekognition/), one of the [AWS AI services](https://aws.amazon.com/machine-learning/ai-services/) for automated image and video analysis with machine learning. It provides **Automated Machine Learning (AutoML)** capability for custom computer vision end-to-end machine learning workflows.

It is suitable for anyone who wants to quickly build a custom computer vision model to classify images, detect objects and scenes unique to their use cases. No machine learning expertise is required.

# **Prerequisites**
For this walkthrough, you should have the following prerequisites:

- **An AWS account** — You can [create a new account](https://portal.aws.amazon.com/billing/signup#/start) if you don’t have one yet.
- **AWS CLI** — You should install or upgrade to the latest [AWS Command Line Interface (AWS CLI) version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).

# **Creating Least Privilege Access IAM User & Policies**
As a [security best practice](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#lock-away-credentials), it is strongly recommended not to use the AWS account root user for any task where it is not required. Instead, create a new IAM (Identity and Access Management) user and grant the required permissions for the IAM user based on the **principle of least privilege** using [identity-based policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html). This adheres to the [IAM best practices](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/identity-and-access-management.html) under the Security Pillar in the [Machine Learning Lens](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/welcome.html) for the [AWS Well-Architected Framework](https://aws.amazon.com/architecture/well-architected/).

In this walkthrough, the new IAM user requires both **Programmatic access** and **AWS Management Console access**.

![](https://miro.medium.com/max/875/1*kEpAwhqbXUTKcktOK6inDg.png)

A new [customer-managed policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#customer-managed-policies) is created to define the set of permissions required for the IAM user. Besides, a [bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html) is also needed for an existing S3 bucket (in this case, my-rekognition-custom-labels-bucket), which is storing the natural flower dataset for access control. This existing bucket can be created by any user other than the new IAM user.

The policy’s definition in JSON format is as shown.
![](https://miro.medium.com/max/875/1*5m2s8CCPiPAbJ7E1jjJ-eQ.png)

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        },
        {
            "Sid": "s3Policies",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketVersions",
                "s3:CreateBucket",
                "s3:GetBucketAcl",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion",
                "s3:GetObjectTagging",
                "s3:GetBucketVersioning",
                "s3:GetObjectVersionTagging",
                "s3:PutBucketCORS",
                "s3:PutLifecycleConfiguration",
                "s3:PutBucketPolicy",
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:PutBucketVersioning",
                "s3:PutObjectVersionTagging"
            ],
            "Resource": "arn:aws:s3:::custom-labels-console*"
        },
        {
            "Sid": "rekognitionPolicies",
            "Effect": "Allow",
            "Action": [
                "rekognition:CreateProject",
                "rekognition:CreateProjectVersion",
                "rekognition:StartProjectVersion",
                "rekognition:StopProjectVersion",
                "rekognition:DescribeProjects",
                "rekognition:DescribeProjectVersions",
                "rekognition:DetectCustomLabels",
                "rekognition:DeleteProject",
                "rekognition:DeleteProjectVersion"
            ],
            "Resource": "*"
        },
        {
            "Sid": "groundTruthPolicies",
            "Effect": "Allow",
            "Action": [
                "groundtruthlabeling:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "s3ExternalBucketPolicies",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketAcl",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion",
                "s3:GetObjectTagging",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-rekognition-custom-labels-bucket/*"
            ]
        }
    ]
}
```
![](https://miro.medium.com/max/875/1*vxJEoYBcQrxpCW6iD_MyZA.png)

```
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AWSRekognitionS3AclBucketRead20191011",
			"Effect": "Allow",
			"Principal": {
				"Service": "rekognition.amazonaws.com"
			},
			"Action": [
				"s3:GetBucketAcl",
				"s3:GetBucketLocation"
			],
			"Resource": "arn:aws:s3:::my-rekognition-custom-labels-bucket"
		},
		{
			"Sid": "AWSRekognitionS3GetBucket20191011",
			"Effect": "Allow",
			"Principal": {
				"Service": "rekognition.amazonaws.com"
			},
			"Action": [
				"s3:GetObject",
				"s3:GetObjectAcl",
				"s3:GetObjectVersion",
				"s3:GetObjectTagging"
			],
			"Resource": "arn:aws:s3:::my-rekognition-custom-labels-bucket/*"
		},
		{
			"Sid": "AWSRekognitionS3ACLBucketWrite20191011",
			"Effect": "Allow",
			"Principal": {
				"Service": "rekognition.amazonaws.com"
			},
			"Action": "s3:GetBucketAcl",
			"Resource": "arn:aws:s3:::my-rekognition-custom-labels-bucket"
		},
		{
			"Sid": "AWSRekognitionS3PutObject20191011",
			"Effect": "Allow",
			"Principal": {
				"Service": "rekognition.amazonaws.com"
			},
			"Action": "s3:PutObject",
			"Resource": "arn:aws:s3:::my-rekognition-custom-labels-bucket/*",
			"Condition": {
				"StringEquals": {
					"s3:x-amz-acl": "bucket-owner-full-control"
				}
			}
		}
	]
}
```
# **Flower Dataset**
We use the [**Oxford Flower 102 dataset**](https://www.kaggle.com/c/oxford-102-flower-pytorch/data?select=flower_data.zip) from the [Oxford 102 Flower PyTorch Kaggle competition](https://www.kaggle.com/c/oxford-102-flower-pytorch/) for building the natural flower classifier using Amazon Rekognition Custom Labels. We use this instead of the [original dataset](http://www.robots.ox.ac.uk/~vgg/data/flowers/102/102flowers.tgz) from the [Visual Geometry Group, University of Oxford](https://www.robots.ox.ac.uk/~vgg/index.html), because it has already been split into *train*, *valid*, *test* datasets, and more importantly, the data has been labelled with respective flower category numbers accordingly for *train* and *valid*.

This dataset has a total of **8,189** flower images, where the *train* split has *6,552* images (**80%**), the *valid* split has *818* images (**10%**), and the *test* split has *819* images (**10%**). The code snippet below helps to convert each of the 102 flower category numbers to their respective flower category name.

```
import os
import json
with open('cat_to_name.json', 'r') as flower_cat:
    data = flower_cat.read()
flower_types = json.loads(data)
for cur_dir_name, new_dir_name in flower_types.items():
    os.rename(cur_dir_name, new_dir_name)
```

The dataset bucket should have the same folder structure, as shown below, with both train and valid folders. Each should have 102 folders beneath where each folder name corresponds to a specific flower category name.

![Flower Dataset Bucket Folder Structure](https://miro.medium.com/max/875/1*k5wiB9Bkbpeb538it-gEOA.png)

# **Creating a New Flower Classifier Project**
After the necessary setup has been completed, you can sign in to the AWS management console as the IAM user. Follow the steps in this [guide](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/gs-step-create-bucket.html) to create your new project for Amazon Rekognition Custom Labels.

# **Creating New Training and Test Datasets**
We create new training and test datasets for the flower classifier project in Amazon Rekognition Custom Labels by [importing images from the S3 bucket](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/cd-s3.html#cd-procedure). It is important to give the dataset a clear and distinctive name to distinguish between different datasets as well as training or test.

For the training dataset, the S3 folder location is set to the S3 *train* folder path as below. Similarly, for the test dataset, the S3 folder location is set to the S3 *valid* folder path.

*s3://my-rekognition-custom-labels-bucket/datasets/oxford_flowers_102/train/*

*s3://my-rekognition-custom-labels-bucket/datasets/oxford_flowers_102/valid/*

```
train
|- alpine sea holly
|  |- image_06969.jpg
|  |- image_06970.jpg
|  |- ...
|- anthurium
|  |- image_01964.jpg
|  |- image_01965.jpg
|  |- ...
|...
```

```
valid
|- alpine sea holly
|  |- image_06977.jpg
|  |- image_06978.jpg
|  |- ...
|- anthurium
|  |- image_01972.jpg
|  |- image_01975.jpg
|  |- ...
|...
```

![Create Training/Test Dataset from Importing Images from S3 Bucket](https://miro.medium.com/max/1250/1*FJnLlNqMy2p5IzyhcA1pHQ.png)

All the images in both training and test datasets are organized into folder names that represent their respective flower category labels. Please make sure to enable **Automatic Labeling** by checking the box as shown above as Amazon Rekognition Custom Labels supports automatic labeling of these images in such structures. This can save a lot of time and effort from manually labeling large image datasets.

You can safely disregard the “*Make sure that your S3 bucket is correctly configured*” message as you should have applied the bucket policy earlier. Please make sure that your bucket name is correct if you use a different name than the one in this example.

![Make sure the S3 Bucket Policy is Configured Correctly](https://miro.medium.com/max/875/1*Et-0BgVorjXdwVtgEGVKgQ.png)

After you create the training and test datasets, you should use the datasets as listed.

![Training and Test Datasets](https://miro.medium.com/max/1250/1*kYH2IvxEwCKUZwTm6CA72g.png)

When you click into either of the datasets, you should find that all the images are labeled accordingly. You can click on any of the labels to inspect the images of that label. You can also search for a label in the search text box on the left.

![Labeled Images for Moon Orchid](https://miro.medium.com/max/1250/1*L_72a9MaScJI3PV-UOPmgw.png)

# **Training New Flower Classifier Model**
You can train a new model in the Amazon Rekognition Custom Labels console by following this [guide](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/tm-console.html). To create a test dataset, you should use the “Choose an existing test dataset” option, as shown below, since it should have been created in the previous section.

The training based on this flower dataset could take more than an hour (approximately 1 hour and 20 minutes in this case) to complete.

![Train Model](https://miro.medium.com/max/1250/1*1zW9ZF81CKdi9dFfdOQjyg.png)

# **Evaluating the Trained Model Performance**
After the flower classifier model is trained, you can review the model performance by accessing the [Evaluation Results](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/tr-console.html) in the console, as shown. You can better understand the metrics for evaluating the model performance from this [guide](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/tr-metrics-use.html). You should be able to achieve similar model performance evaluation results with the same datasets in Amazon Rekognition Custom Labels.

The **Per Label Performance** is a great feature that allows you to analyze the performance metrics at per label level so that it’s faster and easier for you to find out which labels are performing better or poorer than the average.

![Flower Classifier Model Evaluation Results Summary and Per Label Performance](https://miro.medium.com/max/1250/1*IeyXmj52TVu4oWabv5iH1A.png)

Besides, you can also review and filter the results (**True Positive**, **False Positive**, **False Negative**) of the test images to understand where the model is making incorrect predictions. This information helps you to improve your model’s performance by indicating how to change or add images to your training or test dataset.

![Test Results Evaluation Gallery](https://miro.medium.com/max/1250/1*4Sr10Niva_FmlR9DMZP1Kg.png)

![Test Results filtered by False Positive](https://miro.medium.com/max/1250/1*pLBuc2O5p1uUQhv0op31Ig.png)

# **Starting the Flower Classifier Model**
When you are happy with the performance of your trained flower classifier model, you can use it to predict flowers of your choice. Before you can use it, you need to start the model. At the bottom section of the model evaluation results page, there are sample AWS CLI commands on how to start, stop, and analyze flower images with your model. You can refer to this [guide](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/gs-step-start-model.html) for the detailed step to start the model and set up the AWS CLI for the IAM user.

![Use Model with AWS CLI Commands](https://miro.medium.com/max/1250/1*GQ98iPMTepdbdKuTiiRu0w.png)

To start the model, use the AWS CLI command, as shown below. Note that you should change the command line arguments based on your setup or preference. The named profile is specific to the IAM user created for Amazon Rekognition Custom Labels.

```
aws rekognition start-project-version \
  --project-version-arn "MODEL_ARN" \
  --min-inference-units 1 \
  --region us-east-1 \
  --profile customlabels-iam
```

Starting the model takes a while (approximately 15 minutes in this case) to complete. You should see the model status shows as RUNNING in the console, as shown.

![Model Started](https://miro.medium.com/max/1250/1*KJKmNU3UrqxtzALCQ57vMA.png)

# **Classifying with Unseen Flower Images**
After the model is running, you can use it to predict the flower types of images that do not exist in both the training and test datasets to determine how well your model can perform on supported flower types, which it has not seen before. You can use the AWS CLI command below to determine the predicted label of your image.

```
aws rekognition detect-custom-labels \
  --project-version-arn "MODEL_ARN" \
  --image '{"S3Object": {"Bucket": "BUCKET_NAME", "Name": "IMAGE_PATH"}}' \
  --region us-east-1 \
  --profile customlabels-iam
```

Here are some of the prediction results with datasets that are self-taken or independent from both the training and test datasets.

![Roses](https://miro.medium.com/max/836/1*QPtkfVv4IHtze1e4daxWnQ.jpeg)
```
{
    "CustomLabels": [
        {
            "Name": "rose",
            "Confidence": 99.93900299072266
        }
    ]
}
```

![Lotus](https://miro.medium.com/max/668/1*JK8XGlRMt3smxNwQdQEOyw.jpeg)
```
{
    "CustomLabels": [
        {
            "Name": "lotus",
            "Confidence": 99.7560043334961
        }
    ]
}
```

![Moon Orchid](https://miro.medium.com/max/834/1*H_JLiio0U2izEV4Hjm5H7g.jpeg)

```
{
    "CustomLabels": [
        {
            "Name": "moon orchid",
            "Confidence": 98.02899932861328
        }
    ]
}
```

![Hibiscus](https://miro.medium.com/max/861/1*ZJ_Pv1EIuQbAcEwkMLC-Fw.jpeg)

```
{
    "CustomLabels": [
        {
            "Name": "hibiscus",
            "Confidence": 98.11100006103516
        }
    ]
}
```

![Sunflower](https://miro.medium.com/max/669/1*hebIOScyvLDZ7R5lNjTx2g.jpeg)

```
{
    "CustomLabels": [
        {
            "Name": "sunflower",
            "Confidence": 99.86699676513672
        }
    ]
}
```

![Artificial Flower 1](https://miro.medium.com/max/875/1*_GIMAmdSUiNJTEOH1qmJ8w.jpeg)
```
{
    "CustomLabels": []
}
```

![Artificial Flower 2](https://miro.medium.com/max/875/1*z5PzvSwIBmws4WE0nra4lA.jpeg)

```
{
    "CustomLabels": []
}
```

# **Cleaning Up Resources**
You are charged for the amount of time your model is running. If you have finished using the model, you should stop it. You can use the AWS CLI command below to stop the model to avoid unnecessary costs incurred.

You should also delete the Custom Labels project and datasets in the S3 bucket if they are no longer needed to save costs as well.

```
aws rekognition stop-project-version \
  --project-version-arn "MODEL_ARN" \
  --region us-east-1 \
  --profile customlabels-iam
```

Stopping the model is faster than starting the model. It takes approximately 5 minutes in this case. You should see the model status shows STOPPED in the console.

![Model Stopped](https://miro.medium.com/max/1250/1*zVsg9aHltNkpOqW_ZQDv9w.png)

# **Conclusions and Next Steps**
This post shows the complete step-by-step walkthrough to create a natural flower classifier using Amazon Rekognition Custom Labels with AWS best practices based on the AWS Well-Architected Framework. It also shows that you can build a high-performance custom computer vision model with Amazon Rekognition Custom Labels without machine learning expertise.

The model built in this walkthrough has an *F1 score* of **0.997**, which is not easy to achieve for the same dataset if build from scratch even with extensive machine learning expertise. It is also able to perform well on the samples of the unseen natural flowers and is expected not able to predict on the samples of artificial flowers.

If you are interested in building a natural flower classifier from scratch, you might be interested in my post: [**Build, Train and Deploy A Real-World Flower Classifier of 102 Flower Types** — With TensorFlow 2.3, Amazon SageMaker Python SDK 2.x, and Custom SageMaker Training & Serving Docker Containers](https://towardsdatascience.com/build-train-and-deploy-a-real-world-flower-classifier-of-102-flower-types-a90f66d2092a).