optima-ml 0.3.5.dev2
Distributed hyperparameter optimization and input variable selection for artificial neural networks.
Requires Python: >=3.9.0

# OPTIMA: an Optimization Platform for Tuning Input Variables and Model Parameters

OPTIMA is a framework to perform highly parallelized hyperparameter optimization and input variable selection of arbitrary Keras
or Lightning neural networks for supervised learning tasks.

#### Table of Contents
* [Documentation](#documentation)
* [Installation](#installation)
  + [Preconfigured environments](#preconfigured-environments)
    - [Dresden (Barnard / Romeo)](#dresden-barnard-romeo)
  + [Local installation](#local-installation)
* [Usage](#usage)
  + [Overview](#overview)
  + [Running an optimization](#running-an-optimization)
    - [Local](#local)
    - [Cluster](#cluster)
  + [Run-config](#run-config)
    - [General](#general)
    - [Output](#output)
    - [Inputs](#inputs)
    - [Evaluation](#evaluation)
    - [Training](#training)
    - [Hyperparameter optimization](#hyperparameter-optimization)
      * [General](#general-1)
      * [Bayesian optimization (Optuna)](#bayesian-optimization-optuna)
      * [Population Based Training](#population-based-training)
    - [Trial selection](#trial-selection)
    - [Cross-validation](#cross-validation)
    - [Input variable selection](#input-variable-selection)
    - [Lightning](#lightning-1)
    - [Custom Keras objects](#custom-keras-objects)
    - [Cluster setup](#cluster-setup)
  + [Built-in functionality](#built-in-functionality)
    - [Keras](#keras-1)
    - [Inputs](#inputs-1)
    - [Trial selection](#trial-selection-1)
    - [Evaluation](#evaluation-1)
    - [Input variable selection](#input-variable-selection-1)

## Documentation

A quick overview and instructions on how to configure OPTIMA to perform an optimization is given in [usage](#usage).
A more detailed introduction including a description of the inner workings of OPTIMA can be found in chapter 4.2 of
https://inspirehep.net/literature/2707309. Automatically generated API documentation is published at
https://optima-docs.docs.cern.ch/.

## Installation

### Preconfigured environments

#### Dresden (Barnard / Romeo / Capella)

On Barnard, Romeo, and Capella, preconfigured conda environments for both Keras (`OPTIMA_Keras_cpu` / `OPTIMA_Keras_gpu`)
and Lightning (`OPTIMA_lightning_cpu`) are available at `/projects/materie-09/OPTIMA/conda_env`.

1. If you are using `Anaconda` for the first time, load and initialize `Anaconda`:

    ```
    module load release/23.10 Anaconda3/2023.07-2
    conda init
    ```

    This appends a new code block to your `.bashrc`. For later, it is useful to surround
    this in a function. This should look similar to:

    ```bash
    function load_anaconda {
        # >>> conda initialize >>>
        # !! Contents within this block are managed by 'conda init' !!
        __conda_setup="$('/software/rapids/r23.10/Anaconda3/2023.07-2/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
        if [ $? -eq 0 ]; then
            eval "$__conda_setup"
        else
            if [ -f "/software/rapids/r23.10/Anaconda3/2023.07-2/etc/profile.d/conda.sh" ]; then
                . "/software/rapids/r23.10/Anaconda3/2023.07-2/etc/profile.d/conda.sh"
            else
                export PATH="/software/rapids/r23.10/Anaconda3/2023.07-2/bin:$PATH"
            fi
        fi
        unset __conda_setup
        # <<< conda initialize <<<
    }
    ```

    For the changes to take effect, close and re-open your shell (or `source` your
`.bashrc`), and run:

    ```
    load_anaconda
    ```

2. Point `conda` to the existing environments by adding `/projects/materie-09/OPTIMA/conda_env`
to `envs_dirs` in your `.condarc`:

    ```
    conda config --append envs_dirs /projects/materie-09/OPTIMA/conda_env
    ```

    You should now be able to see the environments when running:

    ```
    conda env list
    ```

3. (optional) To display the environment name instead of the path, run:

    ```
    conda config --set env_prompt '({name}) '
    ```

### Local installation

#### Installation with pip

Each tagged version of OPTIMA is published to PyPI. To set up the python environment and install the most recent version
of OPTIMA, run:

- Keras:

    ```
    pip install optima-ml[keras]
    ```

- Lightning (CPU only):

    ```
    pip install optima-ml[lightning] --extra-index-url https://download.pytorch.org/whl/cpu
    ```

- Lightning (with GPU support):

    ```
    pip install optima-ml[lightning]
    ```

In case a different version of OPTIMA that has not been uploaded to PyPI is needed, clone the repository and, from the
repository's root, run:

- Keras:

    ```
    pip install .[keras]
    ```

- Lightning (CPU only):

    ```
    pip install .[lightning] --extra-index-url https://download.pytorch.org/whl/cpu
    ```

- Lightning (with GPU support):

    ```
    pip install .[lightning]
    ```

### Installation with conda

To set up the python environment and install OPTIMA with conda, clone the repository, checkout the desired version of OPTIMA (e.g. the
current master version or a tagged commit), and run:

#### Keras

- Linux:

    ```
    conda env create -n OPTIMA --file conda-requirements.yml
    conda activate OPTIMA
    pip install --no-deps .
    ```

- MacOS:

    ```
    conda env create -n OPTIMA --file conda-requirements-macos.yml
    conda activate OPTIMA
    pip install --no-deps .
    ```

#### Lightning

OPTIMA with Lightning has not yet been tested on MacOS.

- Linux (CPU only):

    ```
    conda env create -n OPTIMA --file conda-requirements_lightning.yml
    conda activate OPTIMA
    pip install --no-deps .
    ```
- Linux (with GPU support):

    ```
    conda env create -n OPTIMA --file conda-requirements_lightning_gpu.yml
    conda activate OPTIMA
    pip install --no-deps .
    ```

## Usage

### Overview

A single optimization run consists of up to three optimization steps: an input variable selection, the main
hyperparameter optimization using Bayesian optimization, and a “fine-tuning” step using Population Based Training.

The input variable selection implements a backwards elimination algorithm to remove uninformative input variables.
Since this approach is based on a trained model that is representative of the final result of the hyperparameter
optimization, the input variable selection is preceded by an initial hyperparameter optimization using the same settings
as the main hyperparameter optimization. Its results are used to perform the input variable selection whose algorithm is
described [here](https://optima-docs.docs.cern.ch/core/variable_optimization.html#OPTIMA.core.variable_optimization.perform_variable_optimization).
Inputs that are found to be uninformative are removed from the dataset for the rest of the optimization.

Both the pre- and the main hyperparameter optimization perform a Bayesian optimization based on
[Optuna's `TPESampler`](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.TPESampler.html).
After a fixed number of trials, the best hyperparameters are determined, corresponding models are retrained using
cross-validation and their performance is evaluated. This evaluation after each hyperparameter optimization is documented
[here](https://optima-docs.docs.cern.ch/core/evaluation.html#OPTIMA.core.evaluation.evaluate_experiment).

The fine-tuning step uses the implementation of [Population Based Training in
Tune](https://docs.ray.io/en/releases-2.5.1/tune/examples/pbt_guide.html). Hyperparameters that do not support mutation
are fixed to the optimized value found during the main optimization step. The subsequent evaluation is identical to the
evaluation of the previous two steps.

Each part of the optimization can be controlled in great detail using a [run-config file](#run-config). Additionally,
OPTIMA provides a full suite of [built-in functions](#built-in-functionality) that can be used to optimize multilayer
perceptrons for classification tasks - no additional configuration required. For other use cases, some (or all) of these
functions can be re-defined in the run-config to modularly replace built-in functionality where needed. The remaining
built-in behavior is left unchanged, reducing the necessary configuration to a minimum.

### Running an optimization

The optimization is started by loading the python environment, executing `optima` and providing the run-config.
Run `optima --help` for an explanation of the available command line arguments.

Depending on the value of the `cluster` argument, the optimization is started locally or sent off as a cluster job.

#### Local

If the `cluster` command line argument is set to `'local'` (the default), the optimization is started locally. Relevant
command line arguments are:

- `config`: Path to the run-config file. (Required)
- `defaults`: Path to an alternative defaults-config. This overwrites the built-in default parameter values. (Default: `None`)
- `cpus`: Total number of CPUs to use. (Default: `1`)
- `gpus`: Total number of GPUs to use. (Default: `0`)
- `mem_per_cpu`: Amount of memory to allocate per CPU (in MB). (Default: `1000`)
- `cpus_per_trial`: Number of CPUs to use per trial. (Default: `1`)
- `gpus_per_trial`: Number of GPUs to use per trial. This can be a fractional number if multiple trials should share a GPU. (Default: `0`)
- `max_pending_trials`: The maximum number of trials that are allowed to be `'pending'`. (Default: `cpus / cpus_per_trial`)
- `temp_dir`: Overwrite Ray's default root temporary directory (`/tmp/ray`). This must be an absolute path.

#### Cluster

If the `cluster` command line argument is set to the name of a built-in cluster or to `'custom'`, the optimization will be
launched as a cluster job. This is done by manually launching a Ray node on each node in the job in order to build a Ray
cluster, to which OPTIMA in then connected. Ray then takes care of distributing all tasks across the cluster. It is
possible to run multiple optimizations in parallel on the same cluster; each job will consequently build its own Ray cluster.
To ensure the communication between Ray nodes is confined to be within one Ray cluster, a file will be created on a shared
file system that keeps track of the running jobs. The location of this file as well as the ports used for the communication
are set in the cluster class that is discussed below.

In order to setup the OPTIMA python environment in the job, a sourcable script is needed. This can, for example, look
like this:

```bash
#!/usr/bin/env bash

source ~/.bashrc
load_anaconda
conda activate OPTIMA_Keras_cpu
```

If the optimization is to be run on an own cluster, i.e. none of the built-in options in `OPTIMA.hardware_configs`,
`cluster` must be set to `'custom'` and a cluster config file containing a `CustomCluster` class that implements the
[abstract cluster base class](https://optima-docs.docs.cern.ch/hardware_configs/common.html#OPTIMA.hardware_configs.common.Cluster)
needs to be provided via the `cluster_config` command line argument. If you are using a Slurm-based cluster, you can
subclass the built-in `SLURMCluster` class; see [Dresden's Barnard cluster](./OPTIMA/hardware_configs/Dresden_Taurus.py#L42)
for reference. At the moment, no such class is available for HTCondor-based clusters.

Additionally, when not running on a Slurm-based cluster, it may be necessary to extend the
[cluster job class](https://optima-docs.docs.cern.ch/hardware_configs/common.html#OPTIMA.hardware_configs.common.ClusterJob)
that contains the job configuration by defining a `ClusterJob` class in the cluster configuration file.

In addition to the arguments listed in [the previous section](#local), the following command line arguments are relevant
when running on a cluster:

- `name`: Specifies the name of the job and the filename of the output log that is created in a `logs` folder in the
execution directory. (Default: `'DNN_optimization'`)
- `cluster`: Specifies which cluster the optimization should run on. (Default: `'local'`)
- `cluster_config`: Path to a cluster configuration file. Only used if `cluster` is set to `'custom'`. (Default: `None`)
- `mem_per_cpu`: Amount of memory to allocate per CPU (in MB). This overwrites the value provided in the cluster-class. (Default: `None`)
- `fixed_ray_node_size`: If provided, each worker node is forced to have the same number of CPUs and GPUs instead of
letting the cluster management software handle the allocation. This is implicitly set when using GPUs. (Default: False)
- `workers`: Sets the number of nodes to allocate. Only used if `fixed_ray_node_size` is set. (Default: `None`)
- `cpus_per_worker`: Sets the number of CPUs to allocate per node. Only used if `fixed_ray_node_size` is set. (Default: `None`)
- `gpus_per_worker`: Sets the number of GPUs to allocate per node. Only used if `fixed_ray_node_size` is set. (Default: `None`)
- `min_cpus_per_ray_node`: Sets the minimum number of CPUs to allocate per node. This is not used if `fixed_ray_node_size`
is set. (Default: `None`)
- `runtime`: The runtime in hours for which the resources should be reserved. (Default: `12.0`)
- `exclude`: Comma separated list of nodes that should be excluded from the job. (Default: `None`)

### Run-config

The run-config file serves as the central repository for all available settings of an optimization run within OPTIMA.
The following sections collect all parameters required by OPTIMA to perform a full optimization. Within the run-config,
it is only necessary to specify parameters that lack default values or whose values are to be altered from their defaults.

In addition to these required parameters, optional parameters (mostly functions and classes) can be defined in the
run-config to overwrite built-in functionality. These are discussed individually [in the next section](#built-in-functionality).

#### General

Especially for testing, it can be useful to only perform the training for a fixed set of hyperparameters without any
optimization. This is possible when setting `perform_variable_opt`, `perform_main_hyperopt` and `perform_PBT_hyperopt`
to `False`. In this case, the `search_space` is assumed to only contain fixed hyperparameter values, and only the model
training and subsequent evaluation is performed.

| Parameter                        | Default value | Explanation                                                                                                                                                                          |
|----------------------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `perform_variable_opt` *(bool)*  | None          | Whether to perform the initial hyperparameter optimization and subsequent input variable selection step.                                                                             |
| `perform_main_hyperopt` *(bool)* | None          | Whether to perform the main hyperparameter optimization step with Optuna.                                                                                                            |
| `perform_PBT_hyperopt` *(bool)*  | None          | Whether to perform the hyperparameter fine-tuning step with PBT.                                                                                                                     |
| `model_type` *(str)*             | `'Keras'`     | The ML library used to build the model. Allowed values are `'Keras'` and `'Lightning'`.                                                                                              |
| `random_seed` *(int)*            | `42`          | Random seed to make the optimization reproducible. Deterministic execution is only guaranteed if model training is not parallelized, i.e. no two trainings can run at the same time. |

#### Output

| Parameter                        | Default value       | Explanation                                                                                                                                                                                             |
|----------------------------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `output_name` *(str)*            | `'OPTIMA_defaults'` | Name of the output directory. Unless `use_exact_name` is `True`, this is prefixed by a string describing which of the three optimization steps was performed.                                           |
| `output_path` *(str)*            | `'DNN_storage'`     | Path to the directory the output folder should be created in. This path can be absolute or relative to the directory of execution. When running on a cluster, this needs to be on a shared file system. |
| `use_exact_name` *(bool)*        | `False`             | If `True`, disables the prefix of the output folder name.                                                                                                                                               |

#### Inputs

Details on how the input data is loaded and preprocessed can be found [here](https://optima-docs.docs.cern.ch/core/inputs.html#OPTIMA.core.inputs.get_experiment_inputs).

| Parameter                                 | Default value | Explanation                                                                                                                                                                                                                                                                                                                           |
|-------------------------------------------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `produce_inputs_plots` *(bool)*           | `True`        | Whether to produce plots of the input data by calling `plot_input_data()`.                                                                                                                                                                                                                                                            |
| `use_testing_dataset` *(bool)*            | `True`        | If `True`, the input data is plot into training, validation and testing datasets. If `False`, it is split into training and validation sets. This affects both the training during hyperparameter optimization as well as the cross-validation during the experiment evaluation.                                                      |
| `fixed_testing_dataset` *(bool)*          | `True`        | If `True`, use the same testing dataset for all folds during cross-validation. If `False`, the testing dataset is rotated like the training and validation sets. Only used if `use_testing_dataset` is `True`.                                                                                                                        |
| `use_eventNums_splitting` *(bool)*        | `False`       | If `True`, use the event numbers provided by `get_inputs()` to perform the splitting of the dataset according to `(event_number + offset) % N == 0`. If `False`, use random splitting.                                                                                                                                                |
| `eventNums_splitting_N` *(int)*           | `5`           | The `N` in `(event_number + offset) % N == 0`. Controls the size of each dataset when using event number splitting. A value of `5` splits the dataset into 5 parts, 1 of which is used for validation, 1 for testing if `use_testing_dataset` is `True`, and the rest for training. Only used if `use_eventNums_splitting` is `True`. |
| `eventNums_splitting_offset_val` *(int)*  | `0`           | Controls which subset is used for validation. All events satisfying `(event_number + eventNums_splitting_offset_val) % N == 0` are used for validation. Only used if `use_eventNums_splitting` is `True`.                                                                                                                             |
| `eventNums_splitting_offset_test` *(int)* | `1`           | Controls which subset is used for testing. All events satisfying `(event_number + eventNums_splitting_offset_test) % N == 0` are used for testing. Only applied if `use_eventNums_splitting` and `use_testing_dataset` are `True`.                                                                                                    |
| `validation_fraction` *(float)*           | `0.2`         | Controls the fraction of events used for validation. Only used if `use_eventNums_splitting` is `False`.                                                                                                                                                                                                                               |
| `test_fraction` *(float)*                 | `0.2`         | Controls the fraction of events used for testing. Only used if `use_eventNums_splitting` is `False` and `use_testing_dataset` is `True`.                                                                                                                                                                                              |
| `max_num_events` *(int)*                  | `np.inf`      | Controls the number of events to be loaded from the dataset. `np.inf` corresponds to all available events.                                                                                                                                                                                                                            |

#### Evaluation

| Parameter                          | Default value | Explanation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `monitor_name` *(str)*             | `'val_loss'`  | Controls which metric is used as the target of the optimization and for early stopping during training. Allowed are native, weighted native, custom and composite metrics as well as all metrics reported by the ML backend during training (e.g. `"val_loss"`). Make sure to add the `"val_"`-prefix when using native, weighted native or custom metrics.                                                                                                                                                                                                                                                                                                                                                                |
| `monitor_op` *(str)*               | `'min'`       | Is the target metric to be minimized (`'min'`) or maximized (`'max'`)?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `native_metrics` *(list)*          | `[]`          | A list of references to a metric class native to the used ML backend. Each entry must be a tuple of the form `("name", (ClassReference, {"kwarg1": kwarg1, ...}))`. Their values are calculated separately on the training and validation datasets and reported as `"train_name" and "val_name". Event weights are not applied.                                                                                                                                                                                                                                                                                                                                                                                            |
| `weighted_native_metrics` *(list)* | `[]`          | A list of references to a metric class native to the used ML backend. Each entry must be a tuple of the form `("name", (ClassReference, {"kwarg1": kwarg1, ...}))`. Their values are calculated separately on the training and validation datasets and reported as `"train_name" and "val_name". Event weights are applied. Currently, weighted metrics are not supported for Lightning (see https://github.com/Lightning-AI/torchmetrics/issues/784).                                                                                                                                                                                                                                                                     |
| `custom_metrics` *(list)*          | `[]`          | A list of custom metrics. Each entry must be a tuple of the form `("name", callable)`. `callable` needs to accept target values, model prediction and sample weights and return a number or a boolean. Their values are calculated at the end of each epoch separately on the training and validation datasets and reported as `"train_name" and "val_name".                                                                                                                                                                                                                                                                                                                                                               |
| `composite_metrics` *(list)*       | `[]`          | A list of composite metrics. These are metrics that combine the values of already existing metrics. Each entry must be a tuple of the form `("name", (metric_name1, metric_name2, ...), callable)` where `callable` accepts the same number of positional arguments as the number of provided metric names and returns a number or a boolean. They are calculated at the end of each epoch and allow all native, weighted native and custom metrics as well as metrics directly reported by the ML backend (e.g. validation loss) as inputs. It is possible to mix training and validation metrics, allowing e.g. to assess the generalization gap. The value of composite metrics is reported as "name" without a prefix. |
| `overtraining_conditions` *(list)* | `[]`          | A list of overtraining conditions. These are special composite metrics that should return `True` when overtraining is detected and `False` otherwise. Each entry must be a tuple of the form `("name", (metric_name1, metric_name2, ...), callable)` where `callable` accepts the same number of positional arguments as the number of provided metric names and returns a boolean. They are calculated at the end of each epoch and allow all native, weighted native, custom and composite metrics as well as metrics directly reported by the ML backend (e.g. validation loss) as inputs. Naturally, it is possible to mix training and validation metrics. The value of overtraining conditions is not reported.      |

#### Training

| Parameter                                | Default value | Explanation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|------------------------------------------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `max_epochs` *(int)*                     | `200`         | The maximal number of epochs until the training is terminated. Early stopping may terminate the training earlier.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `early_stopping_patience` *(int)*        | `6`           | The maximal number of consecutive epochs without an improvement of the target metric before early-stopping the training.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `overtraining_patience` *(int)*          | `6`           | The maximal number of consecutive epochs with overtraining detected (i.e. at least one overtraining condition detected overtraining) before early-stopping the training. This is ignored if no overtraining conditions are defined.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `callbacks` *(list)*                     | `[]`          | A list of references to a callback class native to the used ML backend (i.e. a subclass of `keras.callbacks.Callback` for Keras or `lightning.pytorch.callbacks.Callback` for Lightning. Each entry must be a tuple of the form `(ClassReference, {"kwarg1": kwarg1, ...})`. The callbacks are only used during the training.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `checkpoint_frequency` *(int)*           | `1`           | The number of epochs to wait between consecutive checkpoints for each model. Since this is only useful to restore the training progress if the optimization is interrupted, this should be set so that, on average, a checkpoint is created every few minutes. For the initial and main hyperparameter optimization, the first checkpoint for each trial is created after `checkpoint_frequency` epochs. Since for Population Based Training, a checkpoint must be created after the last epoch in each perturbation interval, the `checkpoint_frequency` is overwritten with the `perturbation_interval`, unless it is divisible by the `checkpointing_frequency`. The first checkpoint will be created after `checkpointing_frequency` epochs if the `burn_in_period` is divisible by the `checkpointing_frequency`, and after `burn_in_period` epochs otherwise. |
| `crossvalidation_for_fixed_hps` *(bool)* | `True`        | If only a fixed set of hyperparameter is to be used (e.g. for testing purposes), this parameter controls if the training of a single model (`False`) or a full cross-validation (`True`) is performed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

#### Hyperparameter optimization

##### General

| Parameter                             | Default value                                                                                                                 | Explanation