aplusml 0.3.1
Conduct usefulness simulations of ML models embedded in workflows
License: Apache 2.0
Requires Python: >=3.10

<div align="center">
  <h1>APLUS ML</h1>
  <div>
    <strong>A</strong> <strong>P</strong>ython <strong>L</strong>ibrary for <strong>U</strong>sefulness <strong>S</strong>imulations of <strong>M</strong>achine <strong>L</strong>earning Models
  </div>
  <div>
    <h4>
      <a href="https://aplusml.readthedocs.io/en/latest/" target="_blank">⚙ Documentation</a> • <a href="https://pypi.org/project/aplusml/" target="_blank">🐍 PyPi</a> • <a href="https://www.sciencedirect.com/science/article/pii/S1532046423000400?via%3Dihub" target="_blank">📝 Paper</a>
    </h4>
    <h4>
      <code>pip install aplusml</code>
    </h4>
  </div>
  <img src="https://github.com/som-shahlab/aplusml/blob/bc3fba49257092f038a862e870b4cae86f9eb659/assets/graphical%20abstract.png" height="400" />
</div>

## Installation

1. Run the following commands to install **APLUS ML**:

```bash
pip install aplusml
```

2. Install **graphviz** by [downloading it here](https://graphviz.org/download/). If you're on Mac with `homebrew`, simply run:
```
brew install graphviz
```

## Usage

Please see the <a href="https://aplusml.readthedocs.io/en/latest/">documentation</a> for usage examples.

Run `tutorials/synthetic_pad.ipynb` to try an example notebook which works out-of-the-box.

This simulates a utility analysis of PAD referral pathways for synthetic PAD patients.

## Motivation

APLUS ML is a simulation framework for conducting usefulness assessments of machine learning models in workflows.

It aims to quantitatively answer the question: *If I use this ML model within this workflow, will the benefits outweigh the costs, and by how much?*

APLUS was originally developed for clinical workflows in healthcare settings, thus all of our examples are healthcare workflows. However, APLUS ML is a broadly applicable library to any workflow that involves a machine learning model making decisions on a stream of datapoints, and we encourage contributors from any domain to use and extend APLUS ML.

## Tutorials

We showcase APLUS on two clinical workflows: 

1. Early detection of peripheral artery disease (PAD)
2. Triaging patients for advanced care planning (ACP) consults

Jupyter notebooks for these use cases can be found in the `tutorials/` folder.

### Early Detection of PAD

The code used to generate the figures in our paper is located in the `tutorials/` directory in `pad.ipynb`. This notebook loads de-identified patient data from Stanford Hospital, which can be provided upon request.

The workflows analyzed can be found in the `workflows/` folder. The doctor-driven workflow is in `pad_doctor.yaml` while the nurse-driven workflow is in `pad_nurse.yaml`

This `tutorials/pad.ipynb` was used to generate the following figures from the APLUS paper:

![PAD Figure 1](assets/pad%20figure%201.png)

![PAD Figure 2](assets/pad%20heatmap.png)

### Triaging Patients for ACP Consults

The code used to replicate the findings of [Jung et al. 2021](https://pubmed.ncbi.nlm.nih.gov/33355350/) can be found in the `tutorials/` directory in `acp_jung_replication.ipynb`. This notebook loads de-identified patient data from Stanford Hospital, which can be provided upon request.

The workflows analyzed can be found in the `workflows/` folder in `acp_jung_replication.yaml`

![ACP Figure](assets/acp%20figure.png)

## Plot Gallery

Some additional example plots that can be generated by APLUS are included below:

![Additional Plots](assets/additional%20plots.png)

## Development

### Installation

```bash
# Download repo
git clone https://github.com/som-shahlab/aplus.git
cd aplus

# Create environment
conda create -n aplus python=3.10 -y
conda activate aplus
pip install poetry && poetry install
```

### Tests

The file `tests/run_tests.py` runs all of the `test[d].py` files in the `tests/` directory. Each `test[d].py` file has a corresponding `test[d].yaml` file that serves as its input.

To run tests:
```
cd tests
python3 run_tests.py
```

### Documentation

We use [Sphinx](https://www.sphinx-doc.org/en/master/) to build the documentation, and host it on [Read the Docs](https://readthedocs.org/).

To build the docs, run:

```bash
# View server
sphinx-autobuild docs/source docs/build/html

# Build for dist
make html
```

### Deploy to PyPi

```bash
poetry build
poetry publish
```


## Citation

```
@article{wornow2023aplus,
  title={APLUS: A Python Library for Usefulness Simulations of Machine Learning Models in Healthcare},
  author={Wornow, Michael and Ross, Elsie Gyang and Callahan, Alison and Shah, Nigam H},
  journal={Journal of Biomedical Informatics},
  pages={104319},
  year={2023},
  publisher={Elsevier}
}
```