flip-evaluator 1.7
A Difference Evaluator for Alternating Images
Requires Python: >=3.8

![Teaser image](images/teaser.png "Teaser image")

# FLIP: A Tool for Visualizing and Communicating Errors in Rendered Images (v1.7)

By
[Pontus Ebelin](https://research.nvidia.com/person/pontus-ebelin)
and
[Tomas Akenine-Möller](https://research.nvidia.com/person/tomas-akenine-m%C3%B6ller),
with
Jim Nilsson,
[Magnus Oskarsson](https://www1.maths.lth.se/matematiklth/personal/magnuso/),
[Kalle Åström](https://www.maths.lu.se/staff/kalleastrom/),
[Mark D. Fairchild](https://www.rit.edu/directory/mdfpph-mark-fairchild),
and
[Peter Shirley](https://research.nvidia.com/person/peter-shirley).

This repository holds implementations of the [LDR-FLIP](https://research.nvidia.com/publication/2020-07_FLIP)
and [HDR-FLIP](https://research.nvidia.com/publication/2021-05_HDR-FLIP) image error metrics.
It also holds code for the FLIP tool, presented in [Ray Tracing Gems II](https://www.realtimerendering.com/raytracinggems/rtg2/index.html).

The changes made for the different versions of FLIP are summarized in the [version list](https://github.com/NVlabs/flip/blob/main/misc/versionList.md).

[A list of papers](https://github.com/NVlabs/flip/blob/main/misc/papersUsingFLIP.md) that use/cite FLIP.

[A note](https://github.com/NVlabs/flip/blob/main/misc/precision.md) about the precision of FLIP.

[An image gallery](https://research.nvidia.com/node/3525) displaying a large quantity of reference/test images and corresponding error maps from
different metrics.

**Note**: since v1.7, the automatic stop exposure in HDR-FLIP is able to handle reference images whose median luminance is 0.

**Note**: since v1.6, the Python version of FLIP can now be installed via `pip install flip-evaluator`.

**Note**: in v1.3, we switched to a *single header* ([FLIP.h](src/cpp/FLIP.h)) for C++/CUDA for easier integration.

# License

Copyright © 2020-2025, NVIDIA Corporation & Affiliates. All rights reserved.

This work is made available under a [BSD 3-Clause License](LICENSE).

The repository distributes code for `tinyexr`, which is subject to a [BSD 3-Clause License](https://github.com/NVlabs/flip/blob/main/misc/LICENSE-third-party.md#bsd-3-clause-license),<br>
and `stb_image`, which is subject to an [MIT License](https://github.com/NVlabs/flip/blob/main/misc/LICENSE-third-party.md#mit-license).

For individual contributions to the project, please confer the [Individual Contributor License Agreement](https://github.com/NVlabs/flip/blob/main/misc/CLA.md).

For business inquiries, please visit our website and submit the form: [NVIDIA Research Licensing](https://www.nvidia.com/en-us/research/inquiries/).

# Simplest Way To Get Started
The simplest way to run FLIP to compare a test image `testImage.png` to a reference image `referenceImage.png` is as follows:
```
pip install flip-evaluator
flip -r referenceImage.png -t testImage.png
```
For more information about the tool's capabilities, try running `flip -h`.

If you wish to use FLIP in your Python or C++ evaluation scripts, please read the next sections.

# Python (API and Tool)
**Setup** (with pip):
```
pip install flip-evaluator
```

**Usage:**<br>

API:<br>
See the example script `src/python/api_example.py`.

Tool:
```
flip --reference reference.{exr|png} --test test.{exr|png} [--options]
```

See the [README](https://github.com/NVlabs/flip/blob/main/src/python/README.md) in the `python` folder and run `flip -h` for further information and usage instructions.

# C++ and CUDA (API and Tool)
**Setup:**

The `src/cpp/FLIP.sln` solution contains one CUDA backend project and one pure C++ backend project.

Compiling the CUDA project requires a CUDA compatible GPU. Instruction on how to install CUDA can be found [here](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html).

Alternatively, a CMake build can be done by creating a build directory in the `src` directory and invoking CMake on the source `cpp` directory (add `--config Release` to build release configuration on Windows):

```
cd src
mkdir build
cd build
cmake ..
cmake --build . [--config Release]
```

CUDA support is enabled via the `FLIP_ENABLE_CUDA`, which can be passed to CMake on the command line with
`-DFLIP_ENABLE_CUDA=ON` or set interactively with `ccmake` or `cmake-gui`.
`FLIP_LIBRARY` option allows to output a library rather than an executable.

**Usage:**<br>

API:<br>
See the [README](https://github.com/NVlabs/flip/blob/main/src/cpp/README.md).

Tool:
```
flip[-cuda].exe --reference reference.{exr|png} --test test.{exr|png} [options]
```

See the [README](https://github.com/NVlabs/flip/blob/main/src/cpp/README.md) in the `src/cpp` folder and run `flip[-cuda].exe -h` for further information and usage instructions.

# PyTorch (Loss Function)
**Setup** (with Anaconda3 or Miniconda):
```
conda create -n flip_dl python numpy matplotlib
conda activate flip_dl
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
conda install -c conda-forge openexr-python
```

**Usage:**

*Remember to activate the* `flip_dl` *environment through* `conda activate flip_dl` *before using the loss function.*

LDR- and HDR-FLIP are implemented as loss modules in `src/pytorch/flip_loss.py`. An example where the loss function is used to train a simple autoencoder is provided in `src/pytorch/train.py`.

See the [README](https://github.com/NVlabs/flip/blob/main/src/pytorch/README.md) in the `pytorch` folder for further information and usage instructions.

# Citation
If your work uses the FLIP tool to find the errors between *low dynamic range* images,
please cite the LDR-FLIP paper:<br>
[Paper](https://research.nvidia.com/publication/2020-07_FLIP) | [BibTeX](https://github.com/NVlabs/flip/blob/main/misc/LDRFLIP.txt)

If it uses the FLIP tool to find the errors between *high dynamic range* images,
instead cite the HDR-FLIP paper:<br>
[Paper](https://research.nvidia.com/publication/2021-05_HDR-FLIP) | [BibTeX](https://github.com/NVlabs/flip/blob/main/misc/HDRFLIP.txt)

Should your work use the FLIP tool in a more general fashion, please cite the Ray Tracing Gems II article:<br>
[Chapter](https://link.springer.com/chapter/10.1007%2F978-1-4842-7185-8_19) | [BibTeX](https://github.com/NVlabs/flip/blob/main/misc/FLIP.txt)

# Acknowledgements
We appreciate the following peoples' contributions to this repository:
Jonathan Granskog, Jacob Munkberg, Jon Hasselgren, Jefferson Amstutz, Alan Wolfe, Killian Herveau, Vinh Truong, Philippe Dagobert, Hannes Hergeth, Matt Pharr, Tizian Zeltner, Jan Honsbrok, Chris Zhang, Wenzel Jakob, Julian Amann, and Xijie Yang.