imsy-htc 0.0.23
Framework for automatic classification and segmentation of hyperspectral images.
License: MIT
Requires Python: >=3.11
Keywords: surgical data science, open surgery, hyperspectral imaging, organ segmentation, semantic scene segmentation, deep learning

<div align="center">
<a href="https://e130-hyperspectal-tissue-classification.s3.dkfz.de/figures/htc_logo.pdf"><img src="https://e130-hyperspectal-tissue-classification.s3.dkfz.de/figures/htc_logo.svg" alt="Logo" width="600" /></a>

[![Python](https://img.shields.io/pypi/pyversions/imsy-htc.svg)](https://pypi.org/project/imsy-htc)
[![PyPI version](https://badge.fury.io/py/imsy-htc.svg)](https://pypi.org/project/imsy-htc)
[![Tests](https://github.com/IMSY-DKFZ/htc/actions/workflows/tests.yml/badge.svg)](https://github.com/IMSY-DKFZ/htc/actions/workflows/tests.yml)

</div>

# Hyperspectral Tissue Classification

This package is a framework for automated tissue classification and segmentation on medical hyperspectral imaging (HSI) data. It contains:

- The implementation of deep learning models to solve supervised classification and segmentation problems for a variety of different input spatial granularities (pixels, superpixels, patches and entire images, cf. figure below) and modalities (RGB data, raw and processed HSI data) from our paper [“Robust deep learning-based semantic organ segmentation in hyperspectral images”](https://doi.org/10.1016/j.media.2022.102488). It is based on [PyTorch](https://pytorch.org/) and [PyTorch Lightning](https://lightning.ai/).
- The implementation of deep learning models for automated sepsis diagnosis and mortality prediciton, as described in our paper ["AI-powered skin spectral imaging enables instant sepsis diagnosis and outcome prediction in critically ill patients"](https://www.science.org/doi/10.1126/sciadv.adw1968). The models are based on skin HSI data at various spatial granularities (median spectra, patches) and across different modalities (RGB images, raw and processed HSI data). We further provide a multimodal approach that combines HSI data with clinical features such as demographics and blood gas analysis parameters.
- Corresponding pretrained models.
- A pipeline to efficiently load and process HSI data, to aggregate deep learning results and to validate and visualize findings.
- Presentation of several solutions to speed up the data loading process (see [Pytorch Conference 2023 poster details](https://github.com/IMSY-DKFZ/htc/tree/main/README.md#-dealing-with-io-bottlenecks-in-high-throughput-model-training) below).

<div align="center">
<a href="https://e130-hyperspectal-tissue-classification.s3.dkfz.de/figures/MIA_model_overview.pdf"><img src="https://e130-hyperspectal-tissue-classification.s3.dkfz.de/figures/MIA_model_overview.svg" alt="Overview of deep learning models in the htc framework, here shown for HSI input." /></a>
</div>

This framework is designed to work on HSI data from the [Tivita](https://diaspective-vision.com/en/) cameras but you can adapt it to different HSI datasets as well. Potential applications include:

- Use our data loading and processing pipeline to easily access image and meta data for any work utilizing Tivita datasets.
- This repository is tightly coupled to work with the public [HeiPorSPECTRAL](https://heiporspectral.org/) and [xeno-spectral](https://spectralverse-heidelberg.org/xeno-spectral) datasets. If you already downloaded the data, you only need to perform the setup steps and then you can directly use the `htc` framework to work on the data (cf. [our tutorials](https://github.com/IMSY-DKFZ/htc/tree/main/README.md#tutorials)).
- Train your own networks and benefit from a pipeline offering e.g. efficient data loading, correct hierarchical aggregation of results and a set of helpful visualizations.
- Use our pretrained models to initialize the weights for your own training.
- Use our pretrained models to generate predictions for your own data.

If you use the `htc` framework, please consider citing the [corresponding papers](https://github.com/IMSY-DKFZ/htc/tree/main/README.md#papers). You can also cite this repository directly via:

<details closed>
<summary>Cite via BibTeX</summary>

```bibtex
@software{sellner_htc_2023,
  author    = {Sellner, Jan and Seidlitz, Silvia},
  publisher = {Zenodo},
  url       = {https://github.com/IMSY-DKFZ/htc},
  date      = {2025-10-25},
  doi       = {10.5281/zenodo.6577614},
  title     = {Hyperspectral Tissue Classification},
  version   = {v0.0.23},
}
```

</details>

## Setup

### Package Installation

This package can be installed via pip:

```bash
pip install imsy-htc
```

This installs all the required dependencies defined in [`requirements.txt`](https://github.com/IMSY-DKFZ/htc/tree/main/dependencies/requirements.txt). The requirements include the default [PyTorch](https://pytorch.org/) version for your system. In case you have special requirements on the PyTorch version (e.g. CUDA version), you need to manually install and build the `imsy-htc` package (see below).

The installation uses the latest versions of the dependencies to not break your existing environment. However, it can happen that a future update of a dependency breaks the installation or introduces incompatibilities. Unfortunately, we cannot guarantee that we will always make a new release which fixes the dependency problems. For this case, we provide a Docker image (see below) where you can work with the last working set of dependencies. This hopefully retains the functionality of this package even in the (far) future.

> &#x26a0;&#xfe0f; This framework was developed and tested using the Ubuntu 24.04+ Linux distribution. Despite we do provide wheels for Windows and macOS as well, they are not tested.

> &#x26a0;&#xfe0f; Network training and inference was conducted using an RTX 4090 GPU with 24 GiB of memory. It should also work with GPUs which have less memory but you may have to adjust some settings (e.g. the batch size).

<details close>
<summary>PyTorch Compatibility and Custom Pytorch Versions</summary>

Our wheels are bound to the PyTorch ABI used during building of the wheel. This usually matches to most recent PyTorch version so if you keep both packages (`imsy-htc` and `torch`) up-to-date, everything should work. In the following table, we list the PyTorch versions which are compatible with the respective `imsy-htc` version.

| `imsy-htc` | `torch` |
| ---------- | ------- |
| 0.0.9      | 1.13    |
| 0.0.10     | 1.13    |
| 0.0.11     | 2.0     |
| 0.0.12     | 2.0     |
| 0.0.13     | 2.1     |
| 0.0.14     | 2.1     |
| 0.0.15     | 2.2     |
| 0.0.15     | 2.3     |
| 0.0.16     | 2.4     |
| 0.0.17     | 2.5     |
| 0.0.18     | 2.5     |
| 0.0.19     | 2.6     |
| 0.0.20     | 2.6     |
| 0.0.21     | 2.7     |
| 0.0.22     | 2.8     |
| 0.0.23     | 2.9     |

However, we do not make explicit version constraints in the dependencies of the `imsy-htc` package because a future version of PyTorch may still work and we don't want to break the installation if it is not necessary.

If something does not work due to a version mismatch or if you have a custom torch version, you usually get an error like

```text
ImportError: /usr/local/lib/python3.12/site-packages/htc/_cpp.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZNK3c105Error4whatEv
```

which indicates a link error between our binary and your PyTorch version. In this case, you need to build the `imsy-htc` package yourself. You can either install the developer version (see below) or you can use the provided source distribution to build `imsy-htc` while installing:

```bash
# Install the PyTorch version which you want
pip install torch
# Install and build our htc package against your previously installed torch package
pip install --no-build-isolation --no-binary "torch,htc" imsy-htc
```

> &#x26a0;&#xfe0f; Building the `imsy-htc` package manually requires that an up-to-date C++ compiler is available on your system, e.g., `g++ --version` should work on Linux.

</details>

<details close>
<summary>Optional Dependencies (<code>imsy-htc[extra]</code>)</summary>

Some requirements are considered optional (e.g. if they are only needed by certain scripts) and you will get an error message if they are needed but unavailable. You can install them via

```bash
pip install --extra-index-url https://read_package:CnzBrgDfKMWS4cxf-r31@git.dkfz.de/api/v4/projects/15/packages/pypi/simple imsy-htc[extra]
```

or by adding the following lines to your `requirements.txt`

```
--extra-index-url https://read_package:CnzBrgDfKMWS4cxf-r31@git.dkfz.de/api/v4/projects/15/packages/pypi/simple
imsy-htc[extra]
```

This installs the optional dependencies defined in [`requirements-extra.txt`](https://github.com/IMSY-DKFZ/htc/tree/main/dependencies/requirements-extra.txt), including for example our Python wrapper for the [challengeR toolkit](https://github.com/wiesenfa/challengeR).

</details>

<details close>
<summary>Docker</summary>

We also provide a Docker setup. As a prerequisite:

- Clone this repository
- Install [Docker](https://docs.docker.com/get-docker/) and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
- Run arbitrary commands inside the container via the provided `run_docker.py` script:

    ```bash
    # Start an interactive python shell inside the container
    python run_docker.py --image-name ghcr.io/imsy-dkfz/imsy-htc ipython

    # Start any Python script inside the container
    python run_docker.py --image-name ghcr.io/imsy-dkfz/imsy-htc python tutorials/website_HeiPorSPECTRAL_example.py

    # Run any htc command inside the container
    python run_docker.py --image-name ghcr.io/imsy-dkfz/imsy-htc htc info

    # Do anything you want inside the container
    python run_docker.py --image-name ghcr.io/imsy-dkfz/imsy-htc bash
    ```

    The example makes use of our prebuilt Docker image from the [GitHub registry](https://github.com/IMSY-DKFZ/htc/pkgs/container/imsy-htc). There are also images for specific versions available. If you want to build the Docker image yourself, you can omit the `--image-name` argument.

All datasets you provided via an environment variable that starts with `PATH_Tivita` will be accessible in your container (you can also check the generated `dependencies/docker-compose*.override.yml` files for details). Per default, the Docker container is meant for small testing and not for development. This is reflected by the fact that all results are stored inside the container and hence will also be deleted after exiting the container. If you want to keep your results, let the environment variable `PATH_HTC_DOCKER_RESULTS` point to the directory where you want to store the results.

</details>

<details close>
<summary>Developer Installation</summary>

If you want to make changes to the package code (which is highly welcome 😉), we recommend to install the `htc` package in editable mode in a separate conda environment:

```bash
# Set up the conda environment
conda create --yes --name htc python=3.13
conda activate htc

# Install the htc package and its requirements
pip install -r dependencies/requirements-dev.txt
pip install --no-build-isolation -e .
```

Before committing any files, please run the static code checks locally:

```bash
git add .
pre-commit run --all-files
```

</details>

### Environment Variables

This framework can be configured via environment variables. Most importantly, we need to know where your data is located (e.g. `PATH_Tivita_HeiPorSPECTRAL`) and where results should be stored (e.g. `PATH_HTC_RESULTS`). For a full list of possible environment variables, please have a look at the documentation of the [`Settings`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/settings.py) class.

> 💡 If you set an environment variable for a dataset path, it is important that the variable name matches the folder name (e.g. the variable name `PATH_Tivita_HeiPorSPECTRAL` matches the dataset path `my/path/HeiPorSPECTRAL` with its folder name `HeiPorSPECTRAL`, whereas the variable name `PATH_Tivita_some_other_name` does not match). Furthermore, the dataset path needs to point to a directory which contains a `data` and an `intermediates` subfolder.

There are several options to set the environment variables. For example:

- You can specify a variable as part of your bash startup script `~/.bashrc` or before running each command:
    ```bash
    PATH_HTC_RESULTS="~/htc/results" htc training --model image --config "models/image/configs/default"
    ```
    However, this might get cumbersome or might not give you the flexibility you need.
- Recommended if you cloned this repository (in contrast to simply installing it via pip): You can create a `.env` file in the repository root and add your variables, for example:

    ```bash
    export PATH_Tivita_HeiPorSPECTRAL=/mnt/nvme_4tb/HeiPorSPECTRAL
    export PATH_HTC_RESULTS=~/htc/results

    # You can also add your own datasets via (the environment variable name must start with PATH_Tivita)
    # export PATH_Tivita_my_dataset=~/htc/Tivita_my_dataset:shortcut=my_shortcut
    # You can then access it via settings.data_dirs.my_shortcut
    ```

- Recommended if you installed the package via pip: You can create user settings for this application. The location is OS-specific. For Linux the location might be at `~/.config/htc/variables.env`. Please run `htc info` upon package installation to retrieve the exact location on your system. The content of the file is of the same format as of the `.env` above.

After setting your environment variables, it is recommended to run `htc info` to check that your variables are correctly registered in the framework.

## Tutorials

A series of [tutorials](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials) can help you get started on the `htc` framework by guiding you through different usage scenarios.

> 💡 The tutorials make use of our public HSI dataset [HeiPorSPECTRAL](https://heiporspectral.org/). If you want to directly run them, please download the dataset first and make it accessible via the environment variable `PATH_Tivita_HeiPorSPECTRAL` as described above.

- As a start, we recommend to take a look at this [general notebook](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/General.ipynb) which showcases the basic functionalities of the `htc` framework. Namely, it demonstrates the usage of the `DataPath` class which is the entry point to load and process HSI data. For example, you will learn how to read HSI cubes, segmentation masks and meta data. Among others, you can use this information to calculate the median spectrum of an organ.
- You want to perform some spectral analysis? Then we have a notebook on [loading and working with median spectra](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/MedianSpectra.ipynb) for you.
- If you want to use our framework with your own dataset, it might be necessary to write a custom `DataPath` class so that you can load and process your images and annotations. We [collected some tips](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/CustomDataPath.md) on how this can be achieved.
- You have some HSI data at hand and want to use one of our pretrained models to generate predictions? Then our [prediction notebook](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/CreatingPredictions.ipynb) has got you covered.
- You want to use our pretrained models to initialize the weights for your own training? See the section about [pretrained models](https://github.com/IMSY-DKFZ/htc/tree/main/README.md#pretrained-models) below for details.
- You want to use our framework to train a network? The [network training notebook](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/network_training/NetworkTraining.ipynb) will show you how to achieve this on the example of a heart and lung segmentation network.
- If you are interested in our technical validation (e.g. because you want to compare your colorchecker images with ours) and need to create a mask to detect the different colorchecker fields, you might find our automatic [colorchecker mask creation pipeline](https://github.com/IMSY-DKFZ/htc/tree/main/htc/utils/ColorcheckerMaskCreation.ipynb) useful.
- If you are coming from our xeno-learning publication and want to know more about the handling of the data, then we have you covered with an extra tutorial on the [xeno-spectral dataset](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/XenoSpectral.ipynb).

We do not have a separate documentation website for our framework yet. However, most of the functions and classes are documented, so feel free to explore the source code or use your favorite IDE to display the documentation. If something does not become clear from the documentation, feel free to open an issue!

## Pretrained Models

This framework gives you access to a variety of pretrained segmentation and classification models. The models will be automatically downloaded, provided you specify the model type (e.g. `image`) and the run folder (e.g. `2022-02-03_22-58-44_generated_default_model_comparison`). It can then be used for example to [create predictions](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/CreatingPredictions.ipynb) on some data or as a baseline for your own training (see example below).

The following table lists all the models you can get for surgical scene segmentation:

<!-- prettier-ignore -->
| model type | modality | class | run folder |
| ----------- | ----------- | ----------- | ----------- |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_projected-malperfusion_rat2pig_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_rat2pig_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_rat2pig_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_rat2pig_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_projected-malperfusion_rat2human_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_rat2human_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_rat2human_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_rat2human_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_projected-malperfusion_pig2rat_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_pig2rat_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_pig2rat_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_pig2rat_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_projected-malperfusion_pig2human_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_pig2human_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_pig2human_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-malperfusion_pig2human_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_projected-ICG_rat2pig_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-ICG_rat2pig_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-ICG_rat2pig_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-ICG_rat2pig_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_projected-ICG_pig2rat_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-ICG_pig2rat_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-ICG_pig2rat_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_projected-ICG_pig2rat_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_joint_pig-p+rat-p2human_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_joint_pig-p+rat-p2human_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_joint_pig-p+rat-p2human_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_joint_pig-p+rat-p2human_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_baseline_rat_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_rat_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_rat_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_rat_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_baseline_pig_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_pig_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_pig_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_pig_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | `2025-03-09_19-38-10_baseline_human_nested-*-2` (outer folds: [0](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_human_nested-0-2.zip), [1](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_human_nested-1-2.zip), [2](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2025-03-09_19-38-10_baseline_human_nested-2-2.zip)) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2023-02-08_14-48-02_organ_transplantation_0.8`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2023-02-08_14-48-02_organ_transplantation_0.8.zip) |
| image | rgb | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2023-01-29_11-31-04_organ_transplantation_0.8_rgb`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2023-01-29_11-31-04_organ_transplantation_0.8_rgb.zip) |
| image | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2022-02-03_22-58-44_generated_default_model_comparison.zip) |
| image | param | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_parameters_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2022-02-03_22-58-44_generated_default_parameters_model_comparison.zip) |
| image | rgb | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_rgb_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/image@2022-02-03_22-58-44_generated_default_rgb_model_comparison.zip) |
| patch | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_64_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/patch@2022-02-03_22-58-44_generated_default_64_model_comparison.zip) |
| patch | param | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_64_parameters_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/patch@2022-02-03_22-58-44_generated_default_64_parameters_model_comparison.zip) |
| patch | rgb | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_64_rgb_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/patch@2022-02-03_22-58-44_generated_default_64_rgb_model_comparison.zip) |
| patch | hsi | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/patch@2022-02-03_22-58-44_generated_default_model_comparison.zip) |
| patch | param | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_parameters_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/patch@2022-02-03_22-58-44_generated_default_parameters_model_comparison.zip) |
| patch | rgb | [`ModelImage`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/image/ModelImage.py) | [`2022-02-03_22-58-44_generated_default_rgb_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/patch@2022-02-03_22-58-44_generated_default_rgb_model_comparison.zip) |
| superpixel_classification | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | [`2022-02-03_22-58-44_generated_default_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/superpixel_classification@2022-02-03_22-58-44_generated_default_model_comparison.zip) |
| superpixel_classification | param | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | [`2022-02-03_22-58-44_generated_default_parameters_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/superpixel_classification@2022-02-03_22-58-44_generated_default_parameters_model_comparison.zip) |
| superpixel_classification | rgb | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | [`2022-02-03_22-58-44_generated_default_rgb_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/superpixel_classification@2022-02-03_22-58-44_generated_default_rgb_model_comparison.zip) |
| pixel | hsi | [`ModelPixel`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/pixel/ModelPixel.py) | [`2022-02-03_22-58-44_generated_default_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/pixel@2022-02-03_22-58-44_generated_default_model_comparison.zip) |
| pixel | param | [`ModelPixelRGB`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/pixel/ModelPixelRGB.py) | [`2022-02-03_22-58-44_generated_default_parameters_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/pixel@2022-02-03_22-58-44_generated_default_parameters_model_comparison.zip) |
| pixel | rgb | [`ModelPixelRGB`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/pixel/ModelPixelRGB.py) | [`2022-02-03_22-58-44_generated_default_rgb_model_comparison`](https://e130-hyperspectal-tissue-classification.s3.dkfz.de/models/pixel@2022-02-03_22-58-44_generated_default_rgb_model_comparison.zip) |

The following table lists all the models you can get for sepsis diagnosis and mortality prediction:

<!-- prettier-ignore -->
| model type | modality | class | run folder |
| ----------- | ----------- | ----------- | ----------- |
| median_pixel | hsi | [`ModelPixel`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/pixel/ModelPixel.py) | `2025-03-07_13-00-00_survival-inclusion_palm_median_nested-*-4_seed-*-2` |
| median_pixel | hsi | [`ModelPixel`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/pixel/ModelPixel.py) | `2025-03-07_13-00-00_survival-inclusion_finger_median_nested-*-4_seed-*-2` |
| median_pixel | hsi | [`ModelPixel`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/pixel/ModelPixel.py) | `2025-03-07_13-00-00_sepsis-inclusion_palm_median_nested-*-4_seed-*-2` |
| median_pixel | hsi | [`ModelPixel`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/pixel/ModelPixel.py) | `2025-03-07_13-00-00_sepsis-inclusion_finger_median_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_palm_stacked_image_nested-*-4_seed-*-2` |
| image | param | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_palm_image_tpi_nested-*-4_seed-*-2` |
| image | rgb | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_palm_image_rgb_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_palm_image_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_palm_image-meta_demographic+vital+BGA+diagnosis+ventilation+catecholamines_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_palm_image-meta_demographic+vital+BGA+diagnosis+ventilation+catecholamines+lab_nested-*-4_seed-*-2` |
| image | param | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_finger_image_tpi_nested-*-4_seed-*-2` |
| image | rgb | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_finger_image_rgb_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_survival-inclusion_finger_image_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_palm_stacked_image_nested-*-4_seed-*-2` |
| image | param | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_palm_image_tpi_nested-*-4_seed-*-2` |
| image | rgb | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_palm_image_rgb_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_palm_image_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_palm_image-meta_demographic+vital+BGA+diagnosis+ventilation+catecholamines_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_palm_image-meta_demographic+vital+BGA+diagnosis+ventilation+catecholamines+lab_nested-*-4_seed-*-2` |
| image | param | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_finger_image_tpi_nested-*-4_seed-*-2` |
| image | rgb | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_finger_image_rgb_nested-*-4_seed-*-2` |
| image | hsi | [`ModelSuperpixelClassification`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/superpixel_classification/ModelSuperpixelClassification.py) | `2025-03-07_13-00-00_sepsis-inclusion_finger_image_nested-*-4_seed-*-2` |

> 💡 The modality `param` refers to stacked tissue parameter images (named TPI in our papers). For the model type `patch`, pretrained models are available for the patch sizes 64 x 64 and 32 x 32 pixels in the case of surgical scene segmentation models. For sepsis diagnosis and mortality prediction models, a patch size of 224 x 224 is utilized. The modality and patch size is not specified when loading a model as it is already characterized by specifying a certain run folder.

> 💡 A wildcard `*` in the run folder name refers to a collection of models (e.g. from nested cross validation). You can use the name as noted in the table to retrieve all models from this collection as list of models or explicitly set the index to only retrieve one specific model from the collection. If you keep the wildcard for creating predictions (see below), all models will be loaded and the final prediction is an ensemble of the output from all individual networks (e.g. 15 networks with 3 outer and 5 inner folds).

> 💡 In the sepsis diagnosis and mortality prediction models, the term `palm_stacked` in the run folder name indicates models trained on combined palm and finger data. In these models, palm and finger patches were concatenated along the spectral dimension.

After successful installation of the `htc` package, you can use any of the pretrained models listed in the table. There are several ways to use them but the general principle is that models are always specified via their `model` and `run_folder`.

### Option 1: Use the models in your own training pipeline

Every model class listed in the table has a static method [`pretrained_model()`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/models/common/HTCModel.py) which you can use to create a model instance and initialize it with the pretrained weights. The model object will be an instance of `torch.nn.Module`. The function has examples for all the different model types but as a teaser consider the following example which loads the pretrained image HSI network:

```python
import torch
from htc import ModelImage, Normalization

run_folder = "2022-02-03_22-58-44_generated_default_model_comparison"  # HSI model
model = ModelImage.pretrained_model(model="image", run_folder=run_folder, n_channels=100, n_classes=19)
input_data = torch.randn(1, 100, 480, 640)  # NCHW
input_data = Normalization(channel_dim=1)(input_data)  # Model expects L1 normalized input
model(input_data).shape
# torch.Size([1, 19, 480, 640])
```

> 💡 Please note that when initializing the weights as in this example, the segmentation head is initialized randomly. Meaningful predictions on your own data can thus not be expected out of the box, but you will have to train the model on your data first.

### Option 2: Use the models to create predictions for your data

The models can be used to predict segmentation masks for your data. The segmentation models automatically sample from your input image according to the selected model spatial granularity (e.g. by creating patches) and the output is always a segmentation mask for an entire image. The set of output classes is determined by the training configuration, e.g. 18 organ classes + background for our semantic models. There are two alternatives for creating predictions:

1. The [`CreatingPredictions`](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/CreatingPredictions.ipynb) notebook shows how to create predictions for all images in a folder (via the `htc inference` command) and how to map the network output to meaningful label names.
2. If you want to compute predictions directly within your code for custom tensors, batches or paths, you can use the [`SinglePredictor`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/model_processing/SinglePredictor.py) class.

### Option 3: Use the models to train a network with the `htc` package

If you are using the `htc` framework to [train your networks](https://github.com/IMSY-DKFZ/htc/tree/main/tutorials/network_training/NetworkTraining.ipynb), you only need to define the model in your configuration:

```json
{
    "model": {
        "pretrained_model": {
            "model": "image",
            "run_folder": "2022-02-03_22-58-44_generated_default_model_comparison"
        }
    }
}
```

This is very similar to option 1 but may be more convenient if you already train with the `htc` framework.

> 💡 We have a [JSON Schema file](https://github.com/IMSY-DKFZ/htc/tree/main/htc/utils/config.schema) which describes the structure of our config files including descriptions of the attributes.

### Prediction Tables

The above options will only make predictions for your data without evaluating those predictions. If you want to compare the predictions against some reference annotations and compute some metric values (e.g., DSC), then there are two possibilities:

1. Use the `htc tables --input-dir YOUR_INPUT_DIR --output-dir YOUR_OUTPUT_DIR --test --metrics DSC --model image --run-folder 2023-02-08_14-48-02_organ_transplantation_0.8` command to create predictions for a set of paths (cf. `htc tables --help` for more information on the arguments). This will save a validation or test table on disk.
2. Similar to the `SinglePredictor` class from before, there is also a [`SinglePredictionTable`](https://github.com/IMSY-DKFZ/htc/tree/main/htc/model_processing/SinglePredictionTable.py) class to compute a metrics table for your data within your own code (the validation or test table will be returned instead of saved to disk). This is useful if you only want to com