Flutter Object Detection App + YOLOV5 Model.
Tags: flutter, yolov5, pytorch, objectdetection
Published: 2023-02-15 15:43:12

Flutter is a popular open-source mobile application development framework that allows developers to create high-performance, cross-platform applications for both Android and iOS devices. With its ease of use and flexibility, Flutter has become a popular choice for developers looking to build mobile applications.

One of the most popular use cases for mobile applications is object detection, where an application can identify and classify objects in images or videos. YOLOv5 is an advanced object detection algorithm that has gained popularity in recent years for its high accuracy and speed.

In this post, we will explore how to integrate YOLOv5 with Flutter to create an object detection application.

### <u>1. Setting Up the Environment:</u>
To get started, you'll need to set up your development environment. You'll need to have Flutter and Python installed on your computer. After that open the Vscode and initialize a new flutter project.

```
flutter create object_detection
```
Wait for the project to be created.
Now, open the https://pub.dev/packages/flutter_pytorch and navigate to Installing tab there you can find the latest package version for the time being the package version is 1.0.1
Run this command:

With Flutter:
 `$ flutter pub add flutter_pytorch`
This will add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):

```
dependencies:
  flutter_pytorch: ^1.0.1
```
Alternatively, your editor might support flutter pub get. Check the docs for your editor to learn more.

Import it
Now in your Dart code, you can use:
```
import 'package:flutter_pytorch/flutter_pytorch.dart';
```

### <u>2. Preparing the Model.</u>
Before you can use YOLOv5 in your Flutter application, you'll need to train the model on your specific dataset. You can use an existing dataset or create your own dataset to train the model.
For this post I am using the pretrained model of yolov5 available on https://github.com/ultralytics/yolov5 as we are performing object detection we need to converts the pretrained model weights to torchscript format.

**classification**

```
import torch
from torch.utils.mobile_optimizer import optimize_for_mobile
model = torch.load('model_scripted.pt',map_location="cpu")
model.eval()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
optimized_traced_model = optimize_for_mobile(traced_script_module)
optimized_traced_model._save_for_lite_interpreter("model.pt")
```
**object detection (yolov5)**

```
!python export.py --weights "the weights of your model" --include torchscript --img 640 --optimize
```

**example**

```
!python export.py --weights yolov5s.pt --include torchscript --img 640 --optimize
```
### <u>3. Creating a basic UI for the App.</u>

Open the lib folder of the app there you will see main.dart file this is the main compile file of dart.
Run the file in debug mode on an emulator or phone connected.

```
emulator -avd "Your emulator name"
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nkq8eewaf6l6jesgdi61.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z1qc5kc2odc8hwwrmpmo.png)

Now run your App through Vscode in debug as it offers quick reload and hot restart.

Your initial app should like the following
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/knr7i2xnrif5m2ts1xqb.png)


**Creating the UI design**
To design the App we will follow a WBS (Work Breakdown Structure) for designing app and integrating the ml model.
Go to the main.dart file and replace all the code with the following:

```
import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'OBJECT DETECTOR',
      debugShowCheckedModeBanner: false,
      theme: ThemeData(
        // This is the theme of your application.
        //
        // Try running your application with "flutter run". You'll see the
        // application has a blue toolbar. Then, without quitting the app, try
        // changing the primarySwatch below to Colors.green and then invoke
        // "hot reload" (press "r" in the console where you ran "flutter run",
        // or simply save your changes to "hot reload" in a Flutter IDE).
        // Notice that the counter didn't reset back to zero; the application
        // is not restarted.
        primarySwatch: Colors.blue,
      ),
    );
  }
}

```
The above code is nothing just a basic class structure of App with all the unnecessory code removed. After reload you will see a black screen like the following:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mx86nszidw5dzour608s.png)
If you are a beginner don't panic, be patient  we are just starting 😂.

Create a new dart file with the name HomeScreen.dart inside the lib folder.
Now if you type stf and press enter this will auto-magically create the relevant structure of code here we are using statefull widget.

```
class HomeScreen extends StatefulWidget {
  const HomeScreen({super.key});

  @override
  State<HomeScreen> createState() => _HomeScreenState();
}

class _HomeScreenState extends State<HomeScreen> {
  @override
  Widget build(BuildContext context) {
    return Container();
  }
}
```
Now create a text inside the Scaffold Class as:

```
 return Scaffold(
      backgroundColor: Colors.white,
      body: Text("Home Screen"),
    );
```
and call this HomeScreen state to your App State as:

```
home: HomeScreen(),
```
Now you can see the App changes to:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uxvpywejsprlbhq1szrd.png)

Now add the following code on HomeScreen.dart to design the base UI of the App.

```
import 'package:flutter/material.dart';
import 'package:flutter/src/widgets/container.dart';
import 'package:flutter/src/widgets/framework.dart';

class HomeScreen extends StatefulWidget {
  const HomeScreen({super.key});

  @override
  State<HomeScreen> createState() => _HomeScreenState();
}

class _HomeScreenState extends State<HomeScreen> {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text("OBJECT DETECTOR APP")),
      backgroundColor: Colors.white,
      body: Center(
          child: Column(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          //Image with Detections....
          //Button to click pic
          ElevatedButton(
            onPressed: () {},
            child: const Icon(Icons.camera),
          )
        ],
      )),
    );
  }
}
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cxygr0t4kox2w8t4hprs.png)
Now the app is Ready to be integerated with the model.

### <u>4. Integrating the Model with Flutter</u>
Once you've trained the model, you can integrate it with your Flutter application. First create a folder of assets with sub folder of model and labels also declare it on your pubspec.yaml file. As shown in the images bellow:

Folder Structure after creating the additional model and labels folder.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dzjaavx1b4da84976sfj.png)

Pubspec.yaml file after adding the assets path:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w7fxj6allelf6yodqtia.png)
Now place the models converted to the models folder and create labels.txt file place your labels there.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5dlp4jcsq1ol2t09kgnu.png)
Or you can download the model and labels from the following link:
https://github.com/AneeqMalik/flutter_pytorch/tree/main/example/assets
Plz star the repository 😊.
After placing the labels and model to their respective folders it is the time we are waiting for integrating the model to our app.

**Integrating the YOLOv5 Model**

- Create the following Variables:

```
File? _imageFile;
  late ModelObjectDetection _objectModel;
  String? _imagePrediction;
  List? _prediction;
  File? _image;
  ImagePicker _picker = ImagePicker();
  bool objectDetection = false;
  List<ResultObjectDetection?> objDetect = [];
```

- Create a Function to load the model into the App:

```
Future loadModel() async {
    String pathObjectDetectionModel = "assets/models/yolov5s.torchscript";
    try {
      _objectModel = await FlutterPytorch.loadObjectDetectionModel(
          //Remeber here 80 value represents number of classes for custom model it will be different don't forget to change this.
          pathObjectDetectionModel, 80, 640, 640,
          labelPath: "assets/labels/labels.txt");
    } catch (e) {
      if (e is PlatformException) {
        print("only supported for android, Error is $e");
      } else {
        print("Error is $e");
      }
    }
  }
```

- Call the load function inside the initState() to load the model as the App Opens:

```
@override
  void initState() {
    // TODO: implement initState
    super.initState();
    loadModel();
  }
```
- Add the following Widgets inside the Column:

```
 body: Center(
          child: Column(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          //Image with Detections....
          Expanded(
            child: Container(
              height: 150,
              width: 300,
              child: objDetect.isNotEmpty
                  ? _image == null
                      ? Text('No image selected.')
                      : _objectModel!.renderBoxesOnImage(_image!, objDetect)
                  : _image == null
                      ? Text('No image selected.')
                      : Image.file(_image!),
            ),
          ),
          Center(
            child: Visibility(
              visible: _imagePrediction != null,
              child: Text("$_imagePrediction"),
            ),
          ),
          //Button to click pic
          ElevatedButton(
            onPressed: () {
              runObjectDetection();
            },
            child: const Icon(Icons.camera),
          )
        ],
      )),
```

- Finally create a object detection function to get the inferences of the image detected:

```
Future runObjectDetection() async {
    //pick an image

    final XFile? image = await _picker.pickImage(
        source: ImageSource.gallery, maxWidth: 200, maxHeight: 200);
    objDetect = await _objectModel.getImagePrediction(
        await File(image!.path).readAsBytes(),
        minimumScore: 0.1,
        IOUThershold: 0.3);
    objDetect.forEach((element) {
      print({
        "score": element?.score,
        "className": element?.className,
        "class": element?.classIndex,
        "rect": {
          "left": element?.rect.left,
          "top": element?.rect.top,
          "width": element?.rect.width,
          "height": element?.rect.height,
          "right": element?.rect.right,
          "bottom": element?.rect.bottom,
        },
      });
    });
    setState(() {
      _image = File(image!.path);
    });
  }
```
**Changing the default sdk version**
When you run the App after the above aditions the app will not compile and will give an issue similiar to the following:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gippih63bz5gbeo98fbk.png)
You need to change the default sdk version from the following file 
ProjectName\object_detection\android\app\build.gradle:  

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d0rbtlm6t2ezfafeug4c.png)
Save the changes and re-run the app build.
Wait for the build to complete it may give some warning but ignore them for the time being.

### <u>5. Testing the Application</u>
To test your application, you can provide an image or video to the application and see how the model detects objects in the media. 

#### Screenshots

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/58oufv1sq11gcweooikh.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9ggcksdvljgafbhvvx90.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3y3dc8k09y5151g5dgxp.png)

🥳🥳🥳🥳The App is Running Fine and giving detections.
 

#### Link to Source Code
https://github.com/AneeqMalik/Flutter-Object-Detector-App-YOLOv5-

#### Additional Resources/Info
https://pub.dev/packages/flutter_pytorch