Image Recognition with CoreML
Tags: coreml, machinelearning, apple, imagerecognition
Published: 2017-09-10 12:37:06

---
title: Image Recognition with CoreML
published: true
description: In this documentation, basic information about image recognition is explained with CoreML. For the sake of clarity, sample coding will be done on the subject.
tags: coreml, machine-learning, apple, image-recognition
---

##Information on entry-level machine learning

In this documentation, basic information about image recognition is explained with CoreML. For the sake of clarity, sample coding will be done on the subject.

![alt text](https://thepracticaldev.s3.amazonaws.com/i/rbw88tathjp9b2x008lp.png "CoreML example app")

##What is CoreML?
Apple's machine learning framework.

> With Core ML, you can integrate trained machine learning models into your app.
![alt text](https://mehmetkoca.gitbooks.io/coreml/content/assets/Screen%20Shot%202017-08-22%20at%2011.00.54.png "CoreML example app")
A trained model is the result of applying a machine learning algorithm to a set of training data. The model makes predictions based on new input data. For example, a model that's been trained on a region's historical house prices may be able to predict a house's price when given the number of bedrooms and bathrooms.
[-Apple Documentation](https://developer.apple.com/documentation/coreml)

**Note**: To use CoreML framework, you must have the minimum version of Xcode 9 or later.

[Github Repo](https://github.com/mehmetkoca/clever-image)

##What is Machine Learning?
Machine Learning (ML) is a branch of **Artificial Intelligence (AI)**. Its purpose is to take any input data (images, text, speech, statistics) then predict the features of behaviors found in the data.
**ML** allows computers, in our case smartphones, to find hidden insights without being explicitly programmed where to look.
Example;
- Face detection
- Facial features detection
- Image Recognition
- Age prediction

input --> Trained Model --> output

**Why is machine learning important?**

ML can perform tasks that we thought only humans could do. It adds a human feeling to your apps. It makes apps feel “Smart”.
Conclusion

- ML is not new (only native ios is new)
- ML field is growing exponentially
- ML can make your apps do more

##Creating User Interface

For a basic application an imageview, a label and a button will suffice.
**Note:** Designed version of the application is available on [Github repo](https://github.com/mehmetkoca/clever-image).

![alt text](https://mehmetkoca.gitbooks.io/coreml/content/assets/Screen%20Shot%202017-08-24%20at%2015.13.23.png "CoreML example app")

##Choosing Photo with UIImagePickerController

First of all, in the file `ViewController.swift`, image view, label and button clicked function are defined.

```swift
@IBOutlet weak var imageView: UIImageView!
@IBOutlet weak var resultLabel: UILabel!

override func viewDidLoad() {
        super.viewDidLoad()
        // ...
}

@IBAction func selectBtnTapped(_ sender: Any) {
        // ...
} 
```
Before implementing the `selectBtnTapped` method, you need to declare `UIImagePickerControllerDelegate` and `UINavigationControllerDelegate` next to `UIViewController` to use Picker View.

```swift
class ViewController: UIViewController, UIImagePickerControllerDelegate, UINavigationControllerDelegate {
...
```

It's ready to implement picker view now.

```swift
@IBAction func selectBtnTapped(_ sender: Any) {
        let picker = UIImagePickerController()
        picker.delegate = self
        picker.allowsEditing = true
        picker.sourceType = .photoLibrary
        self.present(picker, animated: true, completion: nil)
}
```

And then, implement the `didFinishPickingMediaWithInfo` function to complete the photo selection.

```swift
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any]) {
        imageView.image = info[UIImagePickerControllerEditedImage] as? UIImage
        self.dismiss(animated: true, completion: nil)
}
```

Finally, in the `Info.plist` file we create a row named "Privacy - Photo Library Usage Description". In section "Value" you can write the message you want.

![alt text](https://mehmetkoca.gitbooks.io/coreml/content/assets/Screen%20Shot%202017-08-23%20at%2001.03.04.png "CoreML example app")

##Downloading Model and Creating Functions

Download **Places205-GoogleNet** model from [Apple/Machine-Learning](https://developer.apple.com/machine-learning/) page.
Let's `GoogLeNetPLaces.mlmodel` embed into our project.

![alt text](https://mehmetkoca.gitbooks.io/coreml/content/assets/Screen%20Shot%202017-08-24%20at%2014.29.16.png "CoreML example app")

And then, import CoreML and Vision into `ViewController.swift`.

```swift
import UIKit
import CoreML
import Vision
```

Implement the `recognizeImage` function below `didFinishPickingMediaWithInfo`. The function takes 1 parameter as `CIImage` named **image**.

```swift
func recognizeImage(image: CIImage) {
        // ...
}
```

Define **selectedImage** variable of type `CIImage`.

```swift
@IBOutlet weak var imageView: UIImageView!
    @IBOutlet weak var resultLabel: UILabel!

    var selectedImage = CIImage()
```

Convert the image taken as `UIImage` to `CIImage`, and run recognizeImage() defined inside `didFinishPickingMediaWithInfo`.

```swift
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any]) {
        imageView.image = info[UIImagePickerControllerEditedImage] as? UIImage
        self.dismiss(animated: true, completion: nil)

        if let ciImage = CIImage(image: imageView.image!) {
            self.selectedImage = ciImage
        }

        recognizeImage(image: selectedImage)
}
```

##Creating Request with VNCoreMLRequest

First of all, assign "I'm investigating..." to **resultLabel.text**.

```swift
func recognizeImage(image: CIImage) {

        resultLabel.text = "I'm investigating..."
}
```

Assign the return value of `VNCoreMLModel` function to variable **model**.

```swift
func recognizeImage(image: CIImage) {

        resultLabel.text = "I'm investigating..."

        if let model = try? VNCoreMLModel(for: GoogLeNetPlaces().model) {

        }
}
```

Create the **request** with `VNCoreMLRequest` function and send the variable **model** we created above as parameter which is type `model`.

```swift
func recognizeImage(image: CIImage) {

        resultLabel.text = "I'm investigating..."

        if let model = try? VNCoreMLModel(for: GoogLeNetPlaces().model) {
                let request = VNCoreMLRequest(model: model, completionHandler: { (vnrequest, error) in

                })
        }
}
```

Assign the results of type `VNClassificationObservation` to **results**.

```swift
func recognizeImage(image: CIImage) {

        resultLabel.text = "I'm investigating..."

        if let model = try? VNCoreMLModel(for: GoogLeNetPlaces().model) {
                let request = VNCoreMLRequest(model: model, completionHandler: { (vnrequest, error) in
                        if let results = vnrequest.results as? [VNClassificationObservation] {

                        }
                })
        }
}
```

Assign the first result to **topResult**.

```swift 
func recognizeImage(image: CIImage) {

        resultLabel.text = "I'm investigating..."

        if let model = try? VNCoreMLModel(for: GoogLeNetPlaces().model) {
                let request = VNCoreMLRequest(model: model, completionHandler: { (vnrequest, error) in
                        if let results = vnrequest.results as? [VNClassificationObservation] {
                                let topResult = results.first
                        }
                })
        }
}
```

Implement the `DispatchQueue.main.async` method that will work on the background to create an assumption.

```swift 
func recognizeImage(image: CIImage) {

        resultLabel.text = "I'm investigating..."

        if let model = try? VNCoreMLModel(for: GoogLeNetPlaces().model) {
                let request = VNCoreMLRequest(model: model, completionHandler: { (vnrequest, error) in
                        if let results = vnrequest.results as? [VNClassificationObservation] {
                                let topResult = results.first
                                DispatchQueue.main.async {

                                }
                        }
                })
        }
}
```

Compute the confidence rate of the value stored in **topResult** and assign it to **confidenceRate**. And then, for display purposes **resultLabel.text** is used.

```swift
func recognizeImage(image: CIImage) {

        resultLabel.text = "I'm investigating..."

        if let model = try? VNCoreMLModel(for: GoogLeNetPlaces().model) {
                let request = VNCoreMLRequest(model: model, completionHandler: { (vnrequest, error) in
                        if let results = vnrequest.results as? [VNClassificationObservation] {
                                let topResult = results.first
                                DispatchQueue.main.async {
                                        let confidenceRate = (topResult?.confidence)! * 100
                                        self.resultLabel.text = "\(confidenceRate)% it's \(String(describing: topResult?.identifier))"
                                }
                        }
                })
        }
}
```

##Handler with VNImageRequestHandler

Firstly below **model** method, Create **handler** constant with `VNImageRequestHandler`(Option with ciImage).

```swift
let handler = VNImageRequestHandler(ciImage: image)
```

And then, implement `DispatchQueue.global(qos: .userInteractive).async` method that will work on the background as we did before while computing the confidence rate.

![alt text](https://mehmetkoca.gitbooks.io/coreml/content/assets/Screen%20Shot%202017-08-25%20at%2011.16.54.png "CoreML app")

```swift
let handler = VNImageRequestHandler(ciImage: image)
DispatchQueue.global(qos: .userInteractive).async {

}
```

Create do-catch block and process **request** with **handler.perform()**.

```swift
let handler = VNImageRequestHandler(ciImage: image)
DispatchQueue.global(qos: .userInteractive).async {
    do {
            try handler.perform([request])
    } catch {
            print("Err :(")
    }
}
```

it's done. The final state of recognizeImage function:

```swift
func recognizeImage(image: CIImage) {

        resultLabel.text = "I'm investigating..."

        if let model = try? VNCoreMLModel(for: GoogLeNetPlaces().model) {
            let request = VNCoreMLRequest(model: model, completionHandler: { (vnrequest, error) in
                if let results = vnrequest.results as? [VNClassificationObservation] {
                    let topResult = results.first
                    DispatchQueue.main.async {
                        let confidenceRate = (topResult?.confidence)! * 100
                        let rounded = Int (confidenceRate * 100) / 100
                        self.resultLabel.text = "\(rounded)% it's \(topResult?.identifier ?? "Anonymous")"
                    }
                }
            })
            let handler = VNImageRequestHandler(ciImage: image)
            DispatchQueue.global(qos: .userInteractive).async {
                do {
                    try handler.perform([request])
                } catch {
                    print("Err :(")
                }
            }
        }
    }
```
##Final Status and Trying

Simple CoreML application is completed using Places205-GoogLeNet.

![alt text](https://mehmetkoca.gitbooks.io/coreml/content/assets/gitbookCleverImage.png "CoreML app")

[Github Repo](https://github.com/mehmetkoca/clever-image)
[Read on Gitbook](https://www.gitbook.com/book/mehmetkoca/coreml/details)