Serverless APIs for Machine Learning models
Tags: serverless, machinelearning, python, openwhisk
Published: 2019-07-03 13:19:50

---
title: "Serverless APIs for Machine Learning models"
published: true
description: "Converting IBM Model Asset Exchange (MAX) Machine Learning Models into Serverless APIs using IBM Cloud Functions (Apache OpenWhisk)"
tags: serverless, machine learning, python, openwhisk
canonical_url: http://jamesthom.as/blog/2019/07/02/serverless-max-models/
cover_image: https://developer.ibm.com/developer/articles/introduction-to-the-model-asset-exchange-on-ibm-developer/images/exchange-landing-page.png
---

IBM's [Model Asset eXchange](https://developer.ibm.com/exchanges/models/) provides a [curated list](https://developer.ibm.com/exchanges/models/all/) of free Machine Learning models for developers. Models currently published include detecting [emotions](https://developer.ibm.com/exchanges/models/all/max-facial-emotion-classifier/) or [ages](https://developer.ibm.com/exchanges/models/all/max-facial-age-estimator/) in faces from images, [forecasting the weather](https://developer.ibm.com/exchanges/models/all/max-weather-forecaster/), converting [speech to text](https://developer.ibm.com/exchanges/models/all/max-speech-to-text-converter/) and more. Models are pre-trained and ready for use in the cloud.

Models are published as series of [public Docker images](https://hub.docker.com/search?q=codait&type=image). Images automatically expose a HTTP API for model predictions. Documentation in the model repositories explains how to run images locally (using Docker) or deploy to the cloud (using Kubernetes). This got me thinking… 

**Could MAX models be used from serverless functions?** 🤔

Running machine learning models on serverless platforms can take advantage of the horizontal scalability to process large numbers of computationally intensive classification tasks in parallel. Coupled with the serverless pricing structure ("*no charge for idle*"), this can be an extremely cheap and effective way to perform model classifications in the cloud.

**CHALLENGE ACCEPTED!** 🦸‍♂️🦸‍♀️

After a couple days of experimentation, I had worked out an easy way to [automatically expose MAX models as Serverless APIs](https://github.com/jthomas/serverless-max-models) on [IBM Cloud Functions](https://cloud.ibm.com/openwhisk).  🎉🎉🎉 

*I've given instructions below on how to create those APIs from the models using a simple script. If you just want to use the models, follow those instructions. If you are interested in understanding how this works, keep reading as I explain afterwards what I did...*

## Running MAX models on IBM Cloud Functions

[This repository](https://github.com/jthomas/serverless-max-models) contains a [bash script](https://github.com/jthomas/serverless-max-models/blob/master/build.sh) which builds custom Docker runtimes with MAX models for usage on [IBM Cloud Functions](https://cloud.ibm.com/openwhisk). Pushing these images to Docker Hub allows IBM Cloud Functions to use them as [custom runtimes](https://github.com/apache/incubator-openwhisk/blob/master/docs/actions-docker.md). [Web Actions](https://github.com/apache/incubator-openwhisk/blob/master/docs/webactions.md) created from these custom runtime images expose the same Prediction API described in the model documentation. They can be used with no further changes or custom code needed.

### prerequisites

Please follow the links below to set up the following tools before proceeding.

- [Docker](https://www.docker.com/)
- [Docker Hub account](https://hub.docker.com/)
- [IBM Cloud account](https://cloud.ibm.com/registration) 
- [IBM Cloud Functions CLI installed](https://cloud.ibm.com/openwhisk/learn/cli)

**Check out the "[Serverless MAX Models](https://github.com/jthomas/serverless-max-models) repository. Run all the following commands from that folder.**

```
git clone https://github.com/jthomas/serverless-max-models 
cd serverless-max-models 
```

### build custom runtime images

- Set the following environment variables (`MODELS`) with [MAX model names](https://hub.docker.com/search?q=codait&type=image) and run build script.
  - `MODELS`: MAX model names, e.g. `max-facial-emotion-classifier`
  - `USERNAME`: Docker Hub username.

```
MODELS="..." USERNAME="..." ./build.sh
```

This will create Docker images locally with the MAX model names and push to Docker Hub for usage in IBM Cloud Functions. **IBM Cloud Functions only supports public Docker images as custom runtimes.** 

### create actions using custom runtimes

- Create a Web Action using the [custom Docker runtime](https://github.com/apache/incubator-openwhisk/blob/master/docs/actions-docker.md).

```
ibmcloud wsk action create <MODEL_IMAGE> --docker <DOCKERHUB_NAME>/<MODEL_IMAGE> --web true -m 512
```

- Retrieve the Web Action URL (`https://<REGION>.functions.cloud.ibm.com/api/v1/web/<NS>/default/<ACTION>`)

```
ibmcloud wsk action get <MODEL_IMAGE> --url
```

### invoke web action url with prediction api parameters

Use the same API request parameters as defined in the Prediction API specification with the Web Action URL. This will invoke model predictions and return the result as the HTTP response, e.g.

```
curl -F "image=@assets/happy-baby.jpeg" -XPOST <WEB_ACTION_URL>
```

*NOTE: The first invocation after creating an action may incur long cold-start delays due to the platform pulling the remote image into the local registry. Once the image is available in the platform, both further cold and warm invocations will be much faster.*

## Example

Here is an example of creating a serverless API using the `max-facial-emotion-classifier` [MAX model](https://developer.ibm.com/exchanges/models/all/max-facial-emotion-classifier/). Further examples of models which have been tested are available [here](https://github.com/jthomas/serverless-max-models/blob/master/README.md#models). If you encounter problems, please [open an issue](https://github.com/jthomas/serverless-max-models/issues) on Github.

### max-facial-emotion-classifier

- [Facial Emotion Classifier (`max-facial-emotion-classifier`)](https://developer.ibm.com/exchanges/models/all/max-facial-emotion-classifier/)

Start by creating the action using the custom runtime and then retrieve the Web Action URL.

```
$ ibmcloud wsk action create max-facial-emotion-classifier --docker <DOCKERHUB_NAME>/max-facial-emotion-classifier --web true -m 512
ok: created action max-facial-emotion-classifier
$ ibmcloud wsk action get max-facial-emotion-classifier --url
ok: got action max-facial-emotion-classifier
https://<REGION>.functions.cloud.ibm.com/api/v1/web/<NS>/default/max-facial-emotion-classifier
```

According to the [API definition](http://max-facial-emotion-classifier.max.us-south.containers.appdomain.cloud/) for this model, the prediction API expects a form submission with an image file to classify. Using a [sample image](https://github.com/IBM/MAX-Facial-Emotion-Classifier/blob/master/assets/happy-baby.jpeg) from the model repo, the model can be tested using curl.

```
$ curl -F "image=@happy-baby.jpeg" -XPOST https://<REGION>.functions.cloud.ibm.com/api/v1/web/<NS>/default/max-facial-emotion-classifier
```

```json
{
  "status": "ok",
  "predictions": [
    {
      "detection_box": [
        0.15102639296187684,
        0.3828125,
        0.5293255131964809,
        0.5830078125
      ],
      "emotion_predictions": [
        {
          "label_id": "1",
          "label": "happiness",
          "probability": 0.9860254526138306
        },
        ...
      ]
    }
  ]
}
```

#### performance

*Example Invocation Duration (Cold):* ~4.8 seconds

*Example Invocation Duration (Warm):* ~ 800 ms

## How does this work?

### background

Running machine learning classifications using pre-trained models from serverless functions has historically been challenging due to the following reason… 

> Developers do not control runtime environments in (most) serverless cloud platforms. Libraries and dependencies needed by the functions must be provided in the deployment package. Most platforms limit deployment package sizes (~50MB compressed & ~250MB uncompressed).

Machine Learning libraries and models can be much larger than those deployment size limits. This stops them being included in deployment packages. Loading files dynamically during invocations may be possible but incurs extremely long cold-start delays and additional costs.

Fortunately, [IBM Cloud Functions](https://cloud.ibm.com/openwhisk) is based on the open-source serverless project, [Apache OpenWhisk](http://openwhisk.incubator.apache.org/). This platform supports bespoke function runtimes using [custom Docker images](https://github.com/apache/incubator-openwhisk/blob/master/docs/actions-docker.md). Machine learning libraries and models can therefore be provided in custom runtimes. This removes the need to include them in deployment packages or be loaded at runtime.

*Interested in reading other blog posts about using machine learning libraries and toolkits with IBM Cloud Functions? See [these posts](http://jamesthom.as/blog/2017/08/04/large-applications-on-openwhisk/) for [more details](http://jamesthom.as/blog/2018/08/13/serverless-machine-learning-with-tensorflow-dot-js/).*

### MAX model images

IBM's [Model Asset eXchange](https://developer.ibm.com/exchanges/models/all/) publishes Docker images for each model, alongside the pre-trained model files. Images expose a [HTTP API for predictions](https://github.com/IBM/MAX-Text-Sentiment-Classifier#3-use-the-model) using the model on port 5000, built using Python and Flask. [Swagger files](http://max-text-sentiment-classifier.max.us-south.containers.appdomain.cloud/) for the APIs describe the available operations, input parameters and response bodies.

These images use a custom application framework ([maxfw](https://pypi.org/project/maxfw/)), based on Flask, to standardise exposing MAX models as HTTP APIs. This framework handles input parameter validation, response marshalling, CORS support, etc. This allows model runtimes to just implement the prediction API handlers, rather than the entire HTTP application.

Since the framework already handles exposing the model as a HTTP API, I started looking for a way to simulate an external HTTP request coming into the framework. If this was possible, I could trigger this fake request from a Python Web Action to perform the model classification from input parameters. The Web Action would then covert the HTTP response returned into the valid Web Action response parameters.

### flask test client

Reading through the Flask [documentation](http://flask.pocoo.org/docs/1.0/testing/), I came across the perfect solution! 👏👏👏

> Flask provides a way to test your application by exposing the Werkzeug test Client and handling the context locals for you. You can then use that with your favourite testing solution.

This allows application routes to be executed with the [test client](https://werkzeug.palletsprojects.com/en/0.15.x/test/#werkzeug.test.Client), without actually running the HTTP server.

```python
max_app = MAXApp(API_TITLE, API_DESC, API_VERSION)
max_app.add_api(ModelPredictAPI, '/predict')
test_client = max_app.app.test_client()
r = test_client.post('/model/predict', data=content, headers=headers)
```

Using this code within a serverless Python function allows function invocations to trigger the prediction API.  The serverless function only has to convert input parameters to the fake HTTP request and then serialise the response back to JSON.

### python docker action

The custom MAX model runtime image needs to implement the [HTTP API expected](https://github.com/apache/incubator-openwhisk/blob/master/docs/actions-new.md#action-interface) by Apache OpenWhisk. This API is used to instantiate the runtime environment and then pass in invocation parameters on each request. Since the runtime image contains all files and code need to process requests, the `/init` handler becomes a [no-op](https://english.stackexchange.com/questions/25993/what-does-no-op-mean). The `/run` handler converts [Web Action HTTP parameters](https://github.com/apache/incubator-openwhisk/blob/master/docs/webactions.md#http-context) into the fake HTTP request.

Here is the Python script used to proxy incoming Web Actions requests to the framework model service.

```python
from maxfw.core import MAXApp
from api import ModelPredictAPI
from config import API_TITLE, API_DESC, API_VERSION
import json
import base64
from flask import Flask, request, Response

max_app = MAXApp(API_TITLE, API_DESC, API_VERSION)
max_app.add_api(ModelPredictAPI, '/predict')

# Use flask test client to simulate HTTP requests for the prediction APIs
# HTTP request data will come from action invocation parameters, neat huh? :)
test_client = max_app.app.test_client()
app = Flask(__name__)

# This implements the Docker runtime API used by Apache OpenWhisk
# https://github.com/apache/incubator-openwhisk/blob/master/docs/actions-docker.md
# /init is a no-op as everything is provided in the image.
@app.route("/init", methods=['POST'])
def init():
    return ''

# Action invocation requests will be received as the `value` parameter in request body.
# Web Actions provide HTTP request parameters as `__ow_headers` & `__ow_body` parameters.
@app.route("/run", methods=['POST'])
def run():
    body = request.json
    form_body = body['value']['__ow_body']
    headers = body['value']['__ow_headers']

    # binary image content provided as base64 strings
    content = base64.b64decode(form_body)

    # send fake HTTP request to prediction API with invocation data
    r = test_client.post('/model/predict', data=content, headers=headers)
    r_headers = dict((x, y) for x, y in r.headers)

    # binary data must be encoded as base64 strings to return in JSON response
    is_image = r_headers['Content-Type'].startswith('image')
    r_data = base64.b64encode(r.data) if is_image else r.data
    body = r_data.decode("utf-8")

    response = {'headers': r_headers, 'status': r.status_code, 'body': body }
    print (r.status)
    return Response(json.dumps(response), status=200, mimetype='application/json')

app.run(host='0.0.0.0', port=8080)
```

### building into an image

Since the MAX models already exist as public Docker images, those images can be used as base images when building custom runtimes. Those base images handle adding model files and all dependencies needed to execute them into the image.

This is the `Dockerfile` used by the build script to create the custom model image. The `model` parameter refers to the build argument containing the model name.

```bash
ARG model
FROM codait/${model}:latest

ADD openwhisk.py .

EXPOSE 8080

CMD python openwhisk.py
```

This is then used from the following build script to create a custom runtime image for the model.

```bash
#!/bin/bash

set -e -u

for model in $MODELS; do
  echo "Building $model runtime image"
  docker build -t $model --build-arg model=$model .
  echo "Pushing $model to Docker Hub"
  docker tag $model $USERNAME/$model
  docker push $USERNAME/$model
done 
```

Once the image is published to Docker Hub, it can be referenced when creating new Web Actions (using the `—docker` parameter). 😎

```
ibmcloud wsk action create <MODEL_IMAGE> --docker <DOCKERHUB_NAME>/<MODEL_IMAGE> --web true -m 512
```

## Conclusion

IBM's Model Asset eXchange is a curated collection of Machine Learning models, ready to deploy to the cloud for a variety of tasks. All models are available as a series of public Docker images. Models images automatically expose HTTP APIs for classifications.

Documentation in the model repositories explains how to run them locally and deploy using Kubernetes, but what about using on serverless cloud platforms? Serverless platforms are becoming a popular option for deploying Machine Learning models, due to horizontal scalability and cost advantages.

Looking through the source code for the model images, I discovered a mechanism to hook into the custom model framework used to export the model files as HTTP APIs. This allowed me write a simple wrapper script to proxy serverless function invocations to the model prediction APIs. API responses would be serialised back into the Web Action response format. 

Building this script into a new Docker image, using the existing model image as the base image, created a new runtime which could be used on the platform. Web Actions created from this runtime image would automatically expose the same HTTP APIs as the existing image!