doframework 0.2.1
A testing framework for decision optimization model learning algorithms.
License: Apache-2.0
Requires Python: >=3.8

<!--
# Copyright IBM Corporation 2022
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->

# DOFramework

`doframework` is a testing framework for decision-optimization model learning algorithms. Such algorithms learn part or all of a decision-optimization model from data and solve the model to produce a predicted optimal solution. 

`doframework` randomly generates multiple optimization problems (f,O,D,x*) for your algorithm to learn and solve: 
* f is a continuous piece-wise linear function defined over a domain in d-dimensional space (d>1),
* O is a feasibility region in dom(f) defined by linear constraints,
* D = (X,y) is a dataset derived from f,
* x* is the true optimum of f in O (minimum or maximum).

`doframework` feeds your algorithm constraints and data (O,D) and collects its predicted optimum. The algorithm's predicted optimal value can then be compared to the true optimal value f(x*). By comparing the two over multiple randomly generated optimization problems, `doframework` produces a **prediction profile** for your algorithm.

`doframework` integrates with your algorithm (written in Python).

# Design

`doframework` was designed for optimal cloud distribution following an event-driven approach. 

`doframework` was built on top of [ray](https://www.ray.io/ "Ray -- fast and simple distributed computing") for cloud distribution and [rayvens](https://github.com/project-codeflare/rayvens "Rayvens augments Ray with events") for event driven management.

# Requirements

`doframework` was written for Python version >= 3.8.0. 

`doframework` can run either locally or remotely. For optimal performance, run it on a Kubernetes cluster. Cloud configuration is currently available for AWS and IBM Cloud [OpenShift](https://docs.openshift.com/ "RedHat OpenShift Documentation") clusters.

The framework uses storage (local or S3) to interact with simulation products. Configuration is currently available for [AWS](https://aws.amazon.com/s3/ "AWS S3") or [IBM Cloud Object Storage COS](https://www.ibm.com/cloud/object-storage "IBM Cloud Object Storage").

# Install

To run `doframework` locally, install with

```
$ pip install doframework
```

# Configs

Storage specifications are provided in a `configs.yaml`. You'll find examples under `./configs/*`.

The `configs.yaml` includes the list of source and target bucket names (under `buckets`). If necessary, S3 credentials are added under designated fields.

Here is the format of the `configs.yaml` either for local storage
```
local:
    buckets:
        inputs: '<inputs-folder>'
        inputs_dest: '<inputs-dest-folder>'
        objectives: '<objectives-folder>'
        objectives_dest: '<objectives-dest-folder>'
        data: '<data-folder>'
        data_dest: '<data-dest-folder>'
        solutions: '<solutions-folder>'
```
or S3
```
s3:
    buckets:
        inputs: '<inputs-bucket>'
        inputs_dest: '<inputs-dest-bucket>'
        objectives: '<objectives-bucket>'
        objectives_dest: '<objectives-dest-bucket>'
        data: '<data-bucket>'
        data_dest: '<data-dest-bucket>'
        solutions: '<solutions-bucket>'
    aws_secret_access_key: 'xxxx'
    aws_access_key_id: 'xxxx'
    endpoint_url: 'https://xxx.xxx.xxx'
    region: 'xx-xxxx'
    cloud_service_provider: 'aws'
```
Currently, two S3 providers are available under `s3:cloud_service_provider`: either `aws` or `ibm`. The `endpoint_url` is _optional_ for AWS.

**Bucket / folder names must be distinct**.

# Inputs

`input.json` files provide the necessary metadata for the random genration of optimization problems.

`doframework` will run end to end, once `input.json` files are uploaded to `<inputs-bucket>` / `<inputs-folder>`. 

The jupyter notebook `./notebooks/inputs.ipynb` allows you to automatically generate input files and upload them to `<inputs-bucket>`.

Here is an example of an input file (see input samples `input_basic.json` under `./inputs`).


```
{     
    "f": {
        "vertices": {
            "num": 7,
            "range": [[5.0,20.0],[0.0,10.0]],
        },
        "values": {
            "range": [0.0,5.0]
        },
    },
    "omega" : {
        "ratio": 0.8
    },
    "data" : {
        "N": 750,
        "noise": 0.01,
        "policy_num": 2,
        "scale": 0.4
    },
    "input_file_name": "input_basic.json"
}
```

`f:vertices:num`: number of vertices in the piece-wise linear graph of f.<br>
`f:vertices:range`: f domain will be inside this range.<br>
`f:values:range`: range of f values.<br>
`omega:ratio`: vol(O) / vol(dom(f)) >= ratio.<br>
`data:N`: number of data points to sample.<br>
`data:noise`: response variable noise.<br>
`data:policy_num`: number of centers in Gaussian mix distribution of data.<br>
`data:scale`: max STD of Gaussian mix distribution of data (as a ratio of domain diameter).

It's a good idea to start experimenting on low-dimensional problems. 

# User App Integration

Your algorithm will be integrated into `doframework` once it is decorated with `doframework.resolve`. 

A `doframework` experiment runs with `doframework.run()`. The `run()` utility accepts the decorated model and an absolute path to the `configs.yaml`.

Here is an example a user application `module.py`.

```
import doframework as dof

@dof.resolve
def alg(data: np.array, constraints: np.array, **kwargs):
    ...    
    return optimal_arg, optimal_val, regression_model

if __name__ == '__main__':
    
    dof.run(alg, 'configs.yaml', objectives=5, datasets=3, **kwargs)
```

`doframework` provides the following inputs to your algorithm: 

`data`: 2D np.array with features X = data[ : , :-1] and response variable y = data[ : ,-1].<br>
`constraints`: linear constraints as a 2D numpy array A. A data point x satisfies the constraints when A[ : , :-1]*x + A[ : ,-1] <= 0.<br>

It feeds your algorithm additional inputs in kwargs: 

`lower_bound`: lower bound per feature variable.<br>
`upper_bound`: upper bound per feature variable.<br>
`init_value`: optional initial value.<br>

The `run()` utility accepts the arguments:

`objectives`: number of objective targets to generate per input file.<br>
`datasets`: number of datasets to generate per objective target.<br>
`distribute`: True to run distributively, False to run sequentially.<br>
`logger`: True to see `doframework` logs, False otherwise.<br>
`after_idle_for`: stop running when event stream is idle after this many seconds.<br>
`alg_num_cpus`: number of CPUs to dedicate to your algorithm on each optimization task.<br>
`data_num_cpus`: number of CPUs to dedicate to data generation (useful in high dimensions).


# Algorithm Prediction Profile

Once you are done running a `doframework` experiment, run the notebook `notebooks/profile.ipynb`. It will fetch the relevant experiment products from the target buckets and produce the algorithm's prediction profile and prediction probabilities.

`doframework` produces three types of experiment product files:

* `objective.json`: containing information on (f,O,x*) 
* `data.csv`: containing the dataset the algorithm accepts as input
* `solution.json`: containing the algorithm's predicted optimum

See sample files under `./outputs`.

# Kubernetes Cluster

To run `doframework` on a K8S cluster, make sure you are on the cluster's local `kubectl` context. Log into your cluster, if necessary (applicable to OpenShift, see `./doc/openshift.md`).

You can check your local `kubectl` context and change it if necessary with
```
$ kubectl config current-context
$ kubectl config get-contexts
$ kubectl config use-context cluster_name
>> Switched to context "cluster_name".
```

Now `cd` into your project's folder and run the setup bash script `doframework-setup.sh`. The setup script will generate the cluster configuration file `doframework.yaml` in your project's folder. The setup script requires the absolute path to your `configs.yaml`. Running the setup `.sh` script will establish the `ray` cluster. 

```
$ cd <user_project_folder>
$ doframework-setup.sh --configs ~/path/to/configs.yaml
```

You have the option to adapt `doframework.yaml` to fit your application. 

Use the flag `--project-requirements` to specify the absolute path to your `requirements.txt` file. It will be `pip install -r requirements.txt` on your cluster nodes. 

Use the flag `--project-dir` to specify the absolute path to your project. It will be pip installed on your cluster nodes. 
```
$ doframework-setup.sh --configs ~/path/to/configs.yaml --project-requirements <absolute_requirements_path> --project-dir <absolute_project_path>
```

Use the `--skip` flag to skip re-generating the `doframework.yaml`.
```
$ doframework-setup.sh --skip
```
Or, in case you are familiar with `ray`, run instead 
```
$ ray up doframework.yaml --no-config-cache --yes
```
Upload `input.json` file(s) to your `<inputs_bucket>`. Now you can submit your application `module.py` to the cluster
```
$ ray submit doframework.yaml module.py
```

# Ray Cluster

To observe the `ray` dashboard, connect to `http://localhost:8265` in your browser. See `./doc/openshift.md` for OpenShift-specific instructions.

Some useful health-check commands: 

* Check the status of `ray` pods
```
$ kubectl get pods -n ray
```
* Check the status of the `ray` head node
```
$ kubectl describe pod rayvens-cluster-head-xxxxx -n ray
```
* Monitor autoscaling with
```
$ ray exec doframework.yaml 'tail -n 100 -f /tmp/ray/session_latest/logs/monitor*'
```
* Connect to a terminal on the head node
```
$ ray attach doframework.yaml
$ ...
$ exit
```
* Get a remote shell to the cluster manually (find the head node ID with `kubectl describe`)
```
$ kubectl -n ray exec -it rayvens-cluster-head-z97wc -- bash
```

After introducing manual changes to `doframework.yaml`, update with
```
$ ray up doframework.yaml --no-config-cache --yes
```

Shutdown the `ray` cluster with
```
$ ray down -y doframework.yaml
```

# Test

Run the setup bash script `doframework-setup.sh` with the `--example` flag to generate the test script  `doframework_example.py` in your project folder.
```
$ cd <user_project_folder>
$ doframework-setup.sh  --configs ~/path/to/configs.yaml --example
```

To run the test script locally, use
```
$ python doframework_example.py --configs ~/path/to/configs.yaml
```

To run the test script on your K8S cluster, use
```
$ ray submit doframework.yaml doframework_example.py --configs configs.yaml
```
[NOTE: we are using the path to the `configs.yaml` that was mounted on cluster nodes under `$HOME`.]

Make sure to upload input json files to `<inputs-bucket>` / `<inputs-folder>` once you run `doframework_example.py`.