React Native + Tensorflow.js - implementing a model
Tags: reactnative, machinelearning, mobile, javascript
Published: 2022-07-15 11:36:49

Original source: [Medium](https://medium.com/@ferlatti.aldo/react-native-tensorflow-js-implementing-a-model-daad1a2c7f30)

There are three reasons why I decided to write this post:

1. Sometime ago I came upon an [article ](https://towardsdatascience.com/building-a-machine-learning-web-application-using-flask-29fa9ea11dac)about how to implement a machine learning model with React. The article was about implementing a simple Gaussian Naïve Bayes binary classifier made with scikit-learn which ran on a Flask backend, while the front was made in React. Obviously, it is a very needed skill and I recommend everyone to read it. However, I had a different problem. What if my model needs to be loaded onto the device, needs to be mobile compatible, it is more than 200MB in size and is made with _Tensorflow_? The ‘simple’ server solution doesn’t work anymore.
2. A claim made on [VentureBeat ](https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/)says that 87% of data science projects never make it into production. That means that only 1 in 10 project are actually being used. Considering all the money and time (a lot of time) needed for developing a model, the odds are not very motivating. After 9 projects you spend your time and efforts on, end up in some cloud folder (because maybe someday will be used), you start to question if this is the right way and if your next project will also be a waste of time.
3. Lastly, not every company has a data science team to build models and a development team to implement the said models. Sometimes, if you want your model to be used by people, you need to put them out there by yourself or nobody will.

Following these points, I wanted to write about a method to put to use our hardly made and time consuming models out there, in the world, by ourselves.

Here I will not write about building the model (it has already been built) but only about its implementation and use.

There are two paths to choose for mobile development: native code or cross-platform. As the choice of development can vary, so it can the choice for model processing. If you prefer native code, then a tensorflow-lite approach would be a better option, on the other hand, a cross-platform approach like React Native, allows to transfer knowledge from the web development into mobile, consequentially making [TensorFlow.js](https://www.tensorflow.org/js) (tfjs) a good choice.

---

As you probably already guessed, in this article I’ll be using the cross-platform path, therefore _tfjs_ will be used as a central library. For the conversion part we need the python library:

`pip install tensorflowjs`

And because we are trying to implement it with React Native, we need the adapter for the framework:

```
npm i @tensorflow/tfjs
npm i @tensorflow/tfjs-react-native
```

The implementation is done in four steps:

1. Transform the model so it can be loaded onto the device and be used with _tfjs_
2. Load the model
3. Transform the input (image) in a way it can be fed to the model
4. And finally make predictions

---


## Model transformation

Once you trained your model and are satisfied with the result, you save the entire model as a SavedModel format. The SavedModel format is a directory containing a protobuf binary and a TensorFlow checkpoint which can be loaded with tensorflow using the _load_model_ function. But this format is not suitable for mobile and cannot be loaded inside Tensorflow.js library. For that, _tfjs_ has a built-in converter which can convert a SavedModel format into a javascript compatible format (JSON + weights: more about it later).

To convert a saved model, use the following command. Be sure you are inside the root directory, where your model is saved:

```
>tensorflowjs_converter --input_format=tf_saved_model --saved_model_tags=serve --weight_shard_size_bytes=30000000 "path_to_your/model/" "converted_model"
```


What does it do? It is capable to convert from 3 types of models (SavedModel, Frozen Model and from Tensorflow Hub). Because of that, we need to specify what type is the input model (_input format_). The output format is an JSON file with the dataflow graph and weight manifest of the model, together with a collection of binary weights files.
If the _shared size_ is smaller than the total size of the model’s weights, then the weights are sliced in multiple files. However, to load the model with _tfjs_, we need weights in a single file. Therefore, if you put a small shared size, keep in mind that you need to merge the output files into one file. The last two lines represents the input path, or the position of your model, and the output directory path where the generated files will be stored.

After running the above command the result should be a _model.json_ and a group1-shard\of\ binary files (in our case, it should be just one shard of weights).

Before jumping to the application, we need to import all the necessary packages, and more importantly the files we just created with the converter.

{% embed https://gist.github.com/f95507d0b0a9eb235f9036b9b1b1c0d3.git %}


## Loading the model

The next step is to load the model which, thanks to _tensorflow_ and its simple api, is basically a one liner. _Tfjs_allows to load graph models and layered models. Since this is a Keras sequential model, we will load a Layered model with _loadLayersModel_ function. We load the weights and the _json_ in one go and to do so we use the helper from the react-native adapter _bundleResourceIO_

{% embed https://gist.github.com/0f39696b3c1367fab121ea1d64edeb59.git %}

After this, the model is loaded and ready to use.


## Input image transformations

Now that we have our model loaded, we need to feed it data. But before that we need do some transformations so it would be compatible with the input shape. My model is for image classification and require a tensor with a size of an image of 300x300 pixels. The input depends on the model and the training of it, so you need to transform it in the way model learned it before. For this I will transform a local image into base64 encoding and then transform it into a tensor.

{% embed https://gist.github.com/c6b71e202436e09286400ebd26b0f754.git %}


## Make predictions

Just as easy it was to load the model, making a prediction is the same. So yeah, a one liner. The _predict_ function can do a prediction on a batch of images, we only need to split the result based on the batch size.

{% embed https://gist.github.com/00baf245e69a31a0f305b35eadd76873.git %}


## Wrap everything together

The only thing left to do is to call our functions. However, before using any _tfjs_ methods, we need to load the package with **tf.ready()**, only after that we can use the _tensorflow_ package. We export this function so we can call it later from wherever we want in the application.

{% embed https://gist.github.com/95a9c3fe54933dbad5b44bbec099ac0c.git %}

---

**Congratulations!** Now you can run inferences on your mobile device. In this case, the _tfjs_library was used only for loading and predictions, but it also has all the tools for training models. I invite you to experiment with it and let me know if it even makes sense to train a model on a mobile device, and if it does, up to what reasonable point.

---

> _Sometimes it is the people no one can imagine anything of who do the things no one can imagine. ― Alan Turing_