thrift4DL 3.2.0
Thrift for Deep Learning
License: MIT

Thrift4DL
=========

What is it?
-----------

``thrift4DL`` is appreviation of Thrift for Deep Learning which is built
to be batching server in production

Highlights
----------

-  Easy to use: require a little bit knowledge and skills to start a
   Deep Learning server in production mode

-  Scalable: scale easily on multiple GPU and multiple clients without
   worrying about concurrency

Installation
------------

You can install the package from Pypi: ``pip install thrift4DL``

or from source:

::

   git clone https://github.com/vocong25/thrift4DL.git

   python setup.py install

Examples
--------

Before you go further, please take a look at ``example`` folder.

Server Side
^^^^^^^^^^^

::

   class ServerHandler(VisionHandler):
       def get_env(self, gpu_id, mem_fraction):
           """ 
           This is function to initialize environments, import packages, ...
           Parameters:
           ----------
           gpu_id: int or str
               GPU ID in int or str format
           mem_fraction: float
               Memory fraction to occupy in GPU
           """
           os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
           os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu_id)
           import tensorflow as tf
           return tf

       def get_model(self, model_path, env_params):
           tf = env_params
           model = MnistModel(tf, model_path)
           return model

       def preprocess(self, model, input):
           """Preprocess for an input"""
           input = json.loads(input)
           input = decode_image(input['image'])
           input = model.preprocessing(input)
           assert input.shape == (1, 28, 28, 1), ValueError("Wrong input shape")
           return input

       def predict(self, model, input):
           """Given a batch of input to predict"""
           input = np.vstack(input)
           result = model.predict(input)
           results = []
           for val in result:
               pred_num = np.argmax(val)
               pred_score = np.max(val)
               results.append({"pred_num": int(pred_num),
                               "pred_score": float(pred_score)})
           return results


   NUM_MODELS = 1
   server = TModelPoolServer(host='0.0.0.0', port='9090',
                             handler_cls=ServerHandler,
                             model_path='mnist.pb', gpu_ids=[6]*NUM_MODELS,
                             mem_fractions=[0.1]*NUM_MODELS,
                             batch_infer_size=100, batch_group_timeout=1,
                             http_port='8100')
   server.serve()

Clients
^^^^^^^

The framework supports 2 ways to request to the server.

-  Using RESTFul API:

::

   import requests
   import numpy as np
   import json

   def test_api(image_hex, host, port):
       url = 'http://0.0.0.0:8181/v1/predict/'
       js = {"image": image_hex}
       response = requests.post(url, json=js)
       return response

-  TCP/IP:

You can use ``thrift`` to gen your code in your required languages by
using thrift_file in ``thrift4DL/thrift_file/service3.thrift`` and use
it as usual.

Or if you use ``python``, there are available client class written in
``example`` folder.

::

   from thrift4DL.client import VisionClient
   import numpy as np
   from zaailabcorelib.ztools import encode_image, load_image

   # Initialize a client
   client = VisionClient(host='127.0.0.1', port='9090')

   # Load and encode image to hex
   img_arr = load_image('../example/mnist/num5.png')
   img_hex = encode_image(img_arr)

   # Request to server
   js = json.dumps({"image": img_hex})
   result = client.predict(js)

Contributors
------------

-  [ 1 ] congvm - congvm.it@gmail.com