Rpi 4 meets Flutter and Rust
Tags: flutter, rust, raspberrypi, linux
Published: 2020-11-06 16:05:05


At Charlie foxtrot we aim to stay relevant and keep up with new and exciting tech. So how do we do that? We experiment! This time we wanted to capitalize on our in-house competence in Flutter and Rust. Our Flutter expert, Jonathan, had read about Darts foreign function interface (ffi) capabilities and wanted to try it in conjunction with Rust. He asked me if I thought if we could build something that would run on a Raspberry pi (rpi). Without having a clue, I said “YES!” and got to work. The resulting code can be found [here](https://github.com/Charlie-Foxtrot-AB/rutter-pi).

After some debating, we landed in an idea where we would have a Flutter interface that let the user take a photo via the rpi camera module and then display the image in the app. This is a simple app but considering that Flutter don’t have official support for the rpi, this could be quite the challenge. We also decided to build a small (read: very small) library in Rust to communicate with the camera. 

This project has quite a few requirements: 
* [Rust](https://www.rust-lang.org/)
* [Flutter](https://flutter.dev/?gclid=CjwKCAiA4o79BRBvEiwAjteoYAKTfaki-mqcniau_hBmw3-lyF_yJ_KCCg7aeBQONhfKgXj-pRWHaBoCOSgQAvD_BwE&gclsrc=aw.ds)
* [flutter-pi](https://github.com/ardera/flutter-pi)
* [rpi 4 4gb ram](https://thepihut.com/collections/raspberry-pi/products/raspberry-pi-4-model-b)
* an [rpi camera module](https://thepihut.com/products/raspberry-pi-camera-module)
* a touch screen, we went with the [Hyperpixel 4.0](https://shop.pimoroni.com/products/hyperpixel-4?variant=12569485443155). 

## Setup

First thing to do is to set up the Raspberry Pi. In the README of the [flutter-pi](https://github.com/ardera/flutter-pi) repo it's mentioned that [flutter-pi](https://github.com/ardera/flutter-pi) should run without Xserver. Therefore I went with a headless version of [RaspberryOS](https://www.raspberrypi.org/downloads/raspberry-pi-os/). Once you've flashed the os to a ssd card, follow the instructions in the [flutter-pi](https://github.com/ardera/flutter-pi) README and install drivers for your touch screen of choice. Be sure to clone the [engine-binaries](https://github.com/ardera/flutter-pi/tree/engine-binaries) branch of [flutter-pi](https://github.com/ardera/flutter-pi) and place the files in the correct directories aswell.  

On the software side we first want to set up Flutter on our laptop. If you are using a linux distro you may have the option to install and run Flutter via snap. I found that this leads to several issues. Because of that I cloned the Flutter repo from Github and installed it manually.  Where you install Flutter doesn’t really matter as long as you put `/flutter/bin` in your `$PATH`.

Next you want to make sure you are using a Flutter version that’s compatible with [flutter-pi](https://github.com/ardera/flutter-pi). At the time of writing this the compatible version is 1.22.2. You can change Flutter-version by checking out the corresponding branch. You can now test your installation by running `flutter create <name>` followed by `flutter build bundle`. If it builds without errors, everything should be correct. 

We will also need to install Rust so that we can build the small library that lets us talk to the camera. Do this by going over to [the Rust website](https://www.rust-lang.org/tools/install) and follow the instructions. Once Rust is installed we need to add the armv7 target for the rust compiler in order for us to cross compile the code. Do this with: ` rustup component add rust-std-armv7-unknown-linux-gnueabihf`. Now create a Rust project using `cargo new –-lib ffi-test`. The last part (ffi-test) is the name of the project and doesn’t matter, choose a name to your liking. As I mentioned earlier we need to cross compile the code since it’s going to run on the rpi and not the machine we are writing the code on. Create a directory called `.cargo` and place a file in there named `config.toml`. In that file place the following: 
```toml
[build] 
target = "armv7-unknown-linux-gnueabihf"  

[target.armv7-unknown-linux-gnueabihf] 
linker = "arm-linux-gnueabihf-gcc" 
```

The last part after “linker =” is the gcc linker for arm. Install this if you don’t already have it on your machine. Google the correct name of the package for your distro. The name displayed here is the package name on Manjaro. 
We also need to declare, inside the `Cargo.toml`, that we want to build a dynamically linked library (`.so`) like this:
```toml
[lib]
name = "ffi_test"
crate-type = ["cdylib"] #dynamic library
```

The next logical step would be to build a small "Hello world" library in Rust and consume it from Dart/Flutter. Lets create a function that returns a hard coded string when called.
```rust
#[no_mangle]
pub extern fn string_from_rust() -> *const c_char {
    let s = CString::new("Hello from Rust!").unwrap(); // Create a CString from "Hello world"
    let p = s.as_ptr(); // Get a pointer to the underlaying memory for s
    std::mem::forget(s); // Give up the responsibility of cleaning up/freeing s
    p
}
```
Since Dart's ffi mechanisms utilizes the C [ABI](https://en.wikipedia.org/wiki/Application_binary_interface), we need to annotate the Rust function with `#[no_mangle]` in order to preserve the function signatures symbol upon compilation. Since we are compiling this into a `.so` we need to mark the function as `extern` so that it can be linked as any other library. The functions return type is `*const c_char`, which is a raw pointer, basically a raw memory adress. We have to do this since Rust's types are not available in Dart. When we send data over to Dart, Rust can no longer take responsibility for cleaning up that data. Rust no longer have the required information regarding that data's usage and could therefore free the data at an inappropriate time. To build this just run `cargo b`  

Now we want to consume the Rust function from the Dart side of the project. To do that we need the following dependencies:
```dart
import 'dart:ffi' as ffi;
import 'dart:ffi';
import 'dart:typed_data';
import 'package:flutter/material.dart';
import 'package:ffi/ffi.dart';
```

Now we can load and use the Rust library:
```dart
typedef NativeRustStringFromRustFunction = ffi.Pointer<Utf8> Function();

class _MyHomePageState extends State<MyHomePage> {
  _MyHomePageState() {
	  this._dl =
		  ffi.DynamicLibrary.open("libffi_test.so");
	  this._string_from_rust_ffi =
		  _dl.lookupFunction<NativeRustStringFromRustFunction, NativeRustStringFromRustFunction>(
          "string_from_rust");
  }
  int _counter = 0;
  String _message = "Photos taken: 0";
  ffi.DynamicLibrary _dl;
	NativeRustStringFromRustFunction _string_from_rust_ffi;
  void _updateMessage {
    setState(() {
      _counter++;
      var foreignMessage = ffi.Utf8.fromUtf8(_string_from_rust_ffi());
      _message = "'$foreignMessage': '$_counter'";
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(_message),
            Text(
              '$_counter',
              style: Theme.of(context).textTheme.headline4,
            ),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _updateMessage,
        tooltip: 'Increment',
        child: Icon(Icons.add),
      ),
    );
  }
}

```
We first load the `.so` library we compiled from Rust, we look up the function and store it in the state. We then write a small helper function that will call the `_string_from_rust_ffi`-function. Then we have to encode the string returned from Rust to UTF8 using `Utf8.fromUtf8()`.  

In order to test this enable desktop support with:
```bash
$ flutter channel dev
$ flutter upgrade
$ flutter config --enable-linux-desktop
```

Check the config with `flutter devices` and see that your os shows up. Then to run the app run `flutter run -d linux`. You should something like this:  
![](https://dev-to-uploads.s3.amazonaws.com/i/4csorut8sjhfmtf9ydci.jpg)

## Moving over to the Rpi

The main goal in this post is to get Flutter running on the rpi while consuming a Rust library. So far we haven't run anything on the rpi. Lets fix that!  

Flutter does not have any official support for rpi, that's why we need to use the aforementioned [flutter-pi](https://github.com/ardera/flutter-pi). If you have followed the instructions in the repo for [flutter-pi](https://github.com/ardera/flutter-pi) you should have it installed on the rpi. It would be ideal if we could just take the code we built earlier and throw it on to the rpi and run it. Unfortunately that's not the case. The `engine-binaries`branch from the [flutter-pi](https://github.com/ardera/flutter-pi) repo needs to be cloned to the "host" (the computer you are coding on) in order for us to build and compile the Flutter app. Once you've done that go through the build steps in the README of [flutter-pi](https://github.com/ardera/flutter-pi). To save myself time I've saved these as bash scripts. These are listed below.

**build_snapshot.sh**
```bash
#!/bin/bash
~/flutter/bin/cache/dart-sdk/bin/dart \
	~/flutter/bin/cache/dart-sdk/bin/snapshots/frontend_server.dart.snapshot \
	--sdk-root ~/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk_product \
	--target=flutter \
	--aot \
	--tfa \
	-Ddart.vm.product=true \
	--packages .packages \
	--output-dill build/kernel_snapshot.dill \
	--verbose \
	--depfile build/kernel_snapshot.d \
	package:hello_world/main.dart
```

**build_so.sh**
```bash
#!/bin/sh
~/engine-binaries/gen_snapshot_linux_x64 \
  --causal_async_stacks \
  --deterministic \
  --snapshot_kind=app-aot-elf \
  --elf=build/app.so \
  --strip \
  --sim_use_hardfp \
  --no-use-integer-division \
  build/kernel_snapshot.dill
```
**build.sh**
```bash
#!/bin/bash
flutter clean && \
	flutter build bundle --release && \
	./build_snapshot.sh && \
	./build_so.sh \
```
To build for [flutter-pi](https://github.com/ardera/flutter-pi) now run the `build.sh`script.
Upload the files to the rpi using the `upload.sh` script:
```bash
#!/bin/bash
rsync -a --info=progress2 ./build/flutter_assets/ pi@raspberrypi:/home/pi/flutter_hello_world_assets && \
	scp ./build/app.so pi@raspberrypi:/home/pi/flutter_hello_world_assets/app.so
```
To run the application `ssh` into the rpi and run `flutter-pi flutter_hello_world_assets`. If all goes well you should see the "Hello world" app from Flutter and once you press the button the text changes to "Hello from rust.  

## Incorporating the camera
Now that we know how to compile and run Rust and Flutter in conjunction on the rpi we want to be able to control the camera from our app. There might be libraries for controlling the camera written in Dart but since we want to use Rust together with Dart, we're going to do it from Rust.  
I've chosen a library called [rascam](https://github.com/pedrosland/rascam). It's a small library built on top of [mmal-sys](https://github.com/pedrosland/mmal-sys). We can add [rascam](https://github.com/pedrosland/rascam) to our Rust project by adding this to the `Cargo.toml` file:
```toml
[dependencies]
rascam = "*"
```
Now lets take a look at the code for taking an image.

```rust
// ffi-test/src/lib.rs
use std::ffi::CString;
use std::os::raw::c_char;
use rascam::*;

// ...

#[repr(C)]
pub struct ImageBuffer {
    img_ptr: *mut u8,
    len: u32,
}

#[no_mangle]
pub extern fn take_photo() -> *mut ImageBuffer {
    let info = info().unwrap();
    if info.cameras.len() < 1 {
        println!("Found 0 cameras. Exiting");
        // note that this does not run destructors
        ::std::process::exit(1);
    }
    println!("{}", info);
    let mut camera = SimpleCamera::new(info.cameras[0].clone()).unwrap();
    camera.activate().unwrap();
    let mut bytes = camera.take_one().unwrap();
    let len = bytes.len() as u32;
    let ret = bytes.as_mut_ptr();
    std::mem::forget(bytes);
    let mut ib = Box::new(ImageBuffer { img_ptr: ret, len});
    let ret = Box::into_raw(ib);
    std::mem::forget(ret);
    ret
}
```
There is a lot going on here. If youve never ventures in to the unforgiving territory of low level Rust, this might look a little daunting. But fear not! I will explain it all, piece by piece.

On a high level this code solves two problems, take a photo and send it to whoever is calling the function. We will start with the second problem, sending the photo to the caller. Since it's just two values we we want to pass on, one might be tempted to utilize a tuple like `(*mut u8, u32)`. This is however not a viable option since the memory layout of a tuple in Rust is not guaranteed. So we went with a simple struct. The `#[repr(C)]` tells the compiler to treat the struct as a C struct. This means that the fields of the struct will be aligned the same way in memory as we declared them in the code.  We need this because the C ABI being the common grounds for communication between Rust and Dart.  

We start with getting the necessary info for the camera. We use this to check if there even is a camera to begin with. If there isn't we just close the process.
```rust
  let info = info().unwrap();
  if info.cameras.len() < 1 {
      println!("Found 0 cameras. Exiting");
      // note that this does not run destructors
      ::std::process::exit(1);
  }
  println!("{}", info);
```
We then instatiate the camera using the info and activate it.
```rust
 let mut camera = SimpleCamera::new(info.cameras[0].clone()).unwrap();
 camera.activate().unwrap();
```
We take a photo, get the number of bytes, get a pointer to the byte array and tell the compiler to forget about the byte array. Just as with the string earlier this array will be cleaned up by Dart.
```rust
  let mut bytes = camera.take_one().unwrap();
  let len = bytes.len() as u32;
  let ret = bytes.as_mut_ptr();
  std::mem::forget(bytes);
```
Next we create an `ImageBuffer` and place it on the heap by wrapping it in a `Box`. If we didn't place it on the heap and still returned a pointer to it, the `ImageBuffer` would be popped of the stack once the function call returns. That would make it possible for the memory to get overwritten while Dart uses the pointer. We don't want that!
```rust
  let mut ib = Box::new(ImageBuffer { img_ptr: ret, len});
  let ret = Box::into_raw(ib);
  std::mem::forget(ret);
  ret
```
With that everything is well and done on the Rust side but how do we actually use this in Dart?
First will need to declare a type to represent the `ImageBuffer` in Dart.
```dart
class ImageBuffer extends Struct {
  Pointer<Uint8> img_ptr; 
  
  @Uint32()
  int len;
}
```

then we need a type for the actual function providing us the `ImageBuffer` and we need to look it up in the library:
```dart
  typedef NativeRustTakePhotoFunction = ffi.Pointer<ImageBuffer> Function();
  // inside MyHomePageState
  _MyHomePageState() {
	  this._dl =
		  ffi.DynamicLibrary.open("/home/pi/target/debug/libffi_test.so");
	  this._take_photo_ffi =
		  _dl.lookupFunction<NativeRustTakePhotoFunction, NativeRustTakePhotoFunction>(
          "take_photo");
  }
```

We then declare a function to be called on button press:
```dart
  void _takePhoto() {
    setState(() {
      _imageBuffer = _take_photo_ffi().ref; // _imageBuffer is a member of our state
      _counter++;
      _message = "Photos taken: '$_counter'";
    });
  }
```
Once all that is done we can register the function with the button: 
```dart
  // Inside a widget array in a scaffold
  if (_imageBuffer != null) 
    Image.memory(_imageBuffer.img_ptr.asTypedList(_imageBuffer.len)),
```
Unfortunately I couldn't build the Rust library on my Linux machine because of [mmal-sys](https://github.com/pedrosland/mmal-sys). There might be a solution for this. But to build this I:
1. pushed the Rust library code to github 
2. built the flutter bundle on the host, using the scripts
3. uploaded it to the rpi and ran it. 
It then gave me this:
![](https://dev-to-uploads.s3.amazonaws.com/i/eed52p020i2necemrmvc.jpg)

Now you should hopefully have a Flutter app running on a rpi!

## Takeaways
For me the main takeaway is that our approach here might actually be a viable option for resource scarce devices. The possibility of writing a beautiful, interactive application in Flutter and have run in a car infotainment system for example is quite exciting. The possibility to handle the low level stuff with languages such as Rust and C, really opens up this domain for Flutter.
I at least know that I am waiting for official rpi support for Flutter so that I can build something more robust!