microcosm-sagemaker 2023.8.3534
Opinionated machine learning organization and configuration
Requires Python: >=3.6
Keywords: microcosm

# microcosm-sagemaker
Opinionated machine learning with SageMaker

## Usage
For best practices, see
[`cookiecutter-microcosm-sagemaker`](https://github.com/globality-corp/cookiecutter-microcosm-sagemaker).

## Profiling
Make sure `pyinstrument` is installed, either using `pip install pyinstrument` or by installing `microcosm-sagemaker` with `profiling` extra dependencies:

```
pip install -e '.[profiling]'
```

To enable profiling of the app, use the `--profile` flag with `runserver`:

```
runserver --profile
```

The service will log that it is in profiling mode and announce the directory to which it is exporting. Each call to the endpoint will be profiled and its results with be stored in a time-tagged html file in the profiling directory.

## Experiment Tracking
To use `Weights and Biases`, install `microcosm-sagemaker` with `wandb` extra depdency:

```
pip install -e '.[wandb]'
```

To enable experiment tracking in an ML repository:

* Choose the experiment tracking stores for your ML model. Currently, we only support `wandb`. To do so, add `wandb` to `graph.use()` in `app_hooks/train/app.py` and `app_hooks/evaluate/app.py`.

* Add the API key for `wandb` to the environment variables injected by Circle CI into the docker instance, by visiting `https://circleci.com/gh/globality-corp/<MODEL-NAME>/edit#env-vars` and adding `WANDB_API_KEY` as an environment variable.

* `Microcosm-sagemaker` automatically adds the config for the active bundle and its dependents to the `wandb`'s run config.

* To report a static metric:

```
class MyClassifier(Bundle):
    ...

    def fit(self, input_data):
        ...
        self.experiment_metrics.log_static(<metric_name>=<metric_value>)
```

* To report a time-series metric:

```
class MyClassifier(Bundle):
    ...

    def fit(self, input_data):
        ...
        self.experiment_metrics.log_timeseries(
            <metric_name>=<metric_value>,
            step=<step_number>
        )
```

Note that the `step` keyword argument must be provided for logging time-series.

## Artifact Tests

If you want to report your artifact tests to wandb, add the following line to the top of your `conftest.py`. 
For more information on using plugins in pytest, see [here](https://docs.pytest.org/en/6.2.x/plugins.html#requiring-loading-plugins-in-a-test-module-or-conftest-file).

```
pytest_plugins = 'pytest_sagemaker'
```

It should be generated by `globality-build`, but in case it is not, also make sure to run artifact
 tests with `--capture=tee-sys`. This will allow to both capture and show stdout.

## Reproducibility

As recommended [here](https://pytorch.org/docs/stable/notes/randomness.html), we seed Python, 
 Numpy and Pytorch random number generators and force Pytorch operations to be deterministic. See 
`microcosm-sagemaker/random.py` for details.

## Distributed training

To support "distributed" training - with multiple processes (like pytorch `DistributedDataParallel`),
we detect if the current process is a "worker process" (non-master member of a process group).
Worker processes are prevented from communicating with the outside world - writing logs and
saving artifacts.

## Training Bundles on Separate Processes

In some circumstances, it is beneficial for the training of a bundle to take place on a separate 
process. An example of this is when using DDP (or a DDP-related mechanism such as DeepSpeed) for multi-GPU 
training. This training mode may be enbled by setting the `spawn_to_fit` property of the `Bundle` to true
(note that the main process with block while the spawned process runs `fit` -- the purpose is to allow for 
more efficient parallelism in the training of each individual bundle, not for the concurrent training of 
multiple bundles).