README.md

metadata

license: cc-by-4.0

LoRA-Ensemble: Uncertainty Modelling for Self-attention Networks

Michelle Halbheer, Dominik J. Mühlematter, Alexander Becker, Dominik Narnhofer, Helge Aasen, Konrad Schindler and Mehmet Ozgur Turkoglu - 2024

Pretrained models

This repository contains the pretrained models corresponding to the code we released on GitHub. The usage of the models with our pipeline is described on GitHub. This repository only contains the models for our final experiments per dataset, not, however, for all intermediate results.

Citation

If you find our work useful or interesting or use our code, please cite our paper as follows

@misc{
  title = {LoRA-Ensemble: Uncertainty Modelling for Self-attention Networks},
  author = {Halbheer, Michelle and M\"uhlematter, Dominik Jan and Becker, Alexander and Narnhofer, Dominik and Aasen, Helge and Schindler, Konrad and Turkoglu, Mehmet Ozgur}
  year = {2024}
  note = {arXiv: <arxiv code>}
}

CIFAR-100

The table below shows the evaluation results obtained using different methods. Each method was trained five times with varying random seeds.

Method (ViT)	Accuracy	ECE	Settings name*	Model weights*
Single Network	$76.6\pm 0.276.6\backslash pm0.2$	$0.144\pm 0.0010.144\backslash pm0.001$	CIFAR100_settings_explicit	Deep_Ensemble_ViT_base_32_1_members_CIFAR100_settings_explicit<seed>.pt
Single Network with LoRA	$79.6\pm 0.279.6\backslash pm0.2$	$\mathbf{\text{0.014}}\pm 0.003\backslash textbf\{0.014\}\backslash pm0.003$	CIFAR100_settings_LoRA	LoRA_Former_ViT_base_32_1_members_CIFAR100_settings_LoRA<seed>.pt
MC Dropout	$77.1\pm 0.577.1\backslash pm0.5$	$0.055\pm 0.0020.055\backslash pm0.002$	CIFAR100_settings_MCDropout	MCDropout_ViT_base_32_16_members_CIFAR100_settings_MCDropout<seed>.pt
Explicit Ensemble	$\underset{‾}{79.8}\pm 0.2\backslash underline\{79.8\}\backslash pm0.2$	$0.098\pm 0.0010.098\backslash pm0.001$	CIFAR100_settings_explicit	Deep_Ensemble_ViT_base_32_16_members_CIFAR100_settings_explicit<seed>.pt
LoRA-Ensemble	$\mathbf{\text{82.5}}\pm 0.1\backslash textbf\{82.5\}\backslash pm0.1$	$\underset{‾}{0.035}\pm 0.001\backslash underline\{0.035\}\backslash pm0.001$	CIFAR100_settings_LoRA	LoRA_Former_ViT_base_32_16_members_CIFAR100_settings_LoRA<seed>.pt

* Settings and model names are followed by a number in the range 1-5 indicating the used random seed.

HAM10000

The table below shows the evaluation results obtained using different methods. Each method was trained five times with varying random seeds.

Method (ViT)	Accuracy	ECE	Settings name*	Model weights*
Single Network	$84.3\pm 0.584.3\backslash pm0.5$	$0.136\pm 0.0060.136\backslash pm0.006$	HAM10000_settings_explicit	Deep_Ensemble_ViT_base_32_1_members_HAM10000_settings_explicit<seed>.pt
Single Network with LoRA	$83.2\pm 0.783.2\backslash pm0.7$	$0.085\pm 0.0040.085\backslash pm0.004$	HAM10000_settings_LoRA	LoRA_Former_ViT_base_32_1_members_HAM10000_settings_LoRA<seed>.pt
MC Dropout	$83.7\pm 0.483.7\backslash pm0.4$	$\underset{‾}{0.099}\pm 0.007\backslash underline\{0.099\}\backslash pm0.007$	HAM10000_settings_MCDropout	MCDropout_ViT_base_32_16_members_HAM10000_settings_MCDropout<seed>.pt
Explicit Ensemble	$\underset{‾}{85.7}\pm 0.3\backslash underline\{85.7\}\backslash pm0.3$	$0.106\pm 0.0020.106\backslash pm0.002$	HAM10000_settings_explicit	Deep_Ensemble_ViT_base_32_16_members_HAM10000_settings_explicit<seed>.pt
LoRA-Ensemble	$\mathbf{\text{88.0}}\pm 0.2\backslash textbf\{88.0\}\backslash pm0.2$	$\mathbf{\text{0.037}}\pm 0.002\backslash textbf\{0.037\}\backslash pm0.002$	HAM10000_settings_LoRA	LoRA_Former_ViT_base_32_16_members_HAM10000_settings_LoRA<seed>.pt

* Settings and model names are followed by a number in the range 1-5 indicating the used random seed.