qoin: gRPC API for MediaPipe
Tags: mediapipe, grpc, firstpost
Published: 2021-04-18 05:13:21

This article is an English-translated of this blog post.
https://blog.hayashikun.com/entry/2020/12/21/183000

I will introduce a project named [qoin](https://github.com/hayashikun/qoin) to make it easy to use mediapipe detection data.
Also, I'll show some application which uses qoin.

# What is [MediaPipe](https://mediapipe.dev/)?

MediaPipe is an open-source, cross-platform application that detects hands, faces, and objects on the video in real-time using machine learning techniques.

[google/mediapipe](https://github.com/google/mediapipe)

This project is developed by Google and mainly written in C++.
In the MediaPipe, Tensorflow performs the detection.

MediaPipe is indeed interesting; however, it is difficult to extract, save, and use the detected data in other applications.
The only (as long as I confirmed) way was to rewrite code and build by myself.

So, I developed qoin to use detection data easily.

# qoin
qoin is a gRPC server and client application and is written in C++.
It includes the detection part of C++ code from MediaPipe.

![qoin-system](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nowj3vyz227j9s3eau7k.png)

Because MediaPipe communicates internally with Protocol Buffers, I used gRPC, neither REST nor GraphQL.

You can run qoin as a gRPC server or client.
The gRPC server returns messages containing detected data as the response, and the service is named PullStream.
Contrary, the gRPC client sent messages as the request, named PushStream.

You can make an application that uses detection data by just implementing a gRPC server or client following the proto definition.

For now, The Face Mesh and Hand Tracking have been implemented.

![face-mesh](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cdzzy81fn8bwg2i2czu5.png)
![hand-tracking](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t58smmer2869c5prwche.png)

The proto definitions of qoin are here.
https://github.com/hayashikun/qoin/tree/master/qoin/proto

The definitions defined in MediaPipe are also needed, so you have to include these definitions explained as follows.

# Fun applications

## [poin](https://github.com/hayashikun/poin)

This is a desktop application that a pointer moves in the display following hand moving.
It is written in Rust using [conrod](https://github.com/PistonDevelopers/conrod) as the GUI framework.
There is no official gRPC implementation, but we can use such as [tonic](https://github.com/hyperium/tonic).

qoin is running as a server, providing HandTrackingPullStream service.

In the build time, it is looking for the proto definitions of qoin and MediaPipe.
https://github.com/hayashikun/poin/blob/master/build.rs

## [pyoin](https://github.com/hayashikun/pyoin)

This is a Python application that judges the direction of the head, the shape of the hand, and the number of the fingers.

![head-direction](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h35u006qwpzijl9g0xbq.gif)
![rock-paper-scissors](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9nn37cc67fa8k0k5diwu.gif)

qoin is running as a server, providing FaceMeshPullStream service for head-direction and HandTrackingPullStream service for rock-paper-scissors.

The proto definitions of qoin and MediaPipe are included here.
https://github.com/hayashikun/pyoin/blob/master/codegen.py

The shape of the hand and the number of the fingers were evaluated using simple neural networks.
https://github.com/hayashikun/pyoin/blob/master/data/janken.ipynb

You can easily save detection data for training.
https://github.com/hayashikun/pyoin/blob/master/app/sampler/hand_tracking.py

## [qover](https://github.com/hayashikun/qover)
The poin and pyoin work on the same PC as the PC qoin running.
qover is an application that exchanges data across the network.

By using qover, you can run poin and pyoin on another machine from the PC qoin running.

![qover](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/752atnlrqi71iqjz6hkh.png)

As I explained before, qoin runs as both a server and client.
qover receives data from qoin running as a client (PushStream) and sends other applications as a server (PullStrem).

This an example of qover.
![qover](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6xps9ettjwync41kkmfh.gif)
qoin is running on the PC as the client, and the data are sent to the qover deployed on ECS.
qover deliver the data to the web page in the smartphone.

The web page was made using Vue, and the detection and display are written like this.
https://github.com/hayashikun/geekten20/blob/master/src/components/HandTransfer.vue

Because of the CORS, you need to use an envoy proxy or something.
https://github.com/hayashikun/qover/blob/master/envoy/envoy.yaml

# [Bazel](https://bazel.build/)

The MediaPipe and many of Google's projects use Bazel as a build tool, and qoin also uses Bazel.

The dependencies are written in the `WORKSPACE` file.
Most are of MediaPipe.

Some rules such as `cc_proto_library` are defined in both the `@com_google_protobuf` and `@rules_cc` repository, and the build would fail if you load the wrong rules.

# Links
- [MediaPipe](https://mediapipe.dev/)
- [google/mediapipe](https://github.com/google/mediapipe)
- [hayashikun/qoin](https://github.com/hayashikun/qoin)
- [hayashikun/poin](https://github.com/hayashikun/poin)
- [hayashikun/pyoin](https://github.com/hayashikun/pyoin)
- [hayashikun/qover](https://github.com/hayashikun/qover)
- [hayashikun/geekten20](https://github.com/hayashikun/geekten20)
- [Bazel](https://bazel.build/)