PETALface Model Card

Project Page | Paper (ArXiv) | Code

Introduction

PETALface is the first work which uses image-quality adaptive LoRA layers for low-resolution face recgonition. The main contributions of our work are:

We introduce the use of the LoRA-based PETL technique to adapt large pre-trained face-recognition models to low-resolution datasets.
We propose an image-quality-based weighting of LoRA modules to create separate proxy encoders for high-resolution and low-resolution data, ensuring effective extraction of embeddings for face recognition.
We demonstrate the superiority of PETALface in adapting to low-resolution datasets, outperforming other state-of-the-art models on low-resolution benchmarks while maintaining performance on high-resolution and mixed-quality datasets.

Training Framework

Overview of the proposed PETALface approach: We include an additional trainable module in linear layers present in attention layers and the final feature projection MLP. The trainable module is highlighted on the right. Specifically, we add two LoRA layers, where the weightage α is decided based on the input-image quality, computed using an off-the-shelf image quality assessment network (IQA).

Usage

The pre-trained weights can be downloaded directly from this repository or using python:

from huggingface_hub import hf_hub_download

# Finetuned Weights

# The filename "swin_arcface_webface4m_tinyface" indicates that the model has a swin bakcbone and pretraind
# on webface4m dataset with arcface loss function and finetuned on tinyface.
hf_hub_download(repo_id="kartiknarayan/PETALface", filename="swin_arcface_webface4m_tinyface/model.pt", local_dir="./weights")
hf_hub_download(repo_id="kartiknarayan/PETALface", filename="swin_cosface_webface4m_tinyface/model.pt", local_dir="./weights")
hf_hub_download(repo_id="kartiknarayan/PETALface", filename="swin_cosface_webface4m_briar/model.pt", local_dir="./weights")
hf_hub_download(repo_id="kartiknarayan/PETALface", filename="swin_cosface_webface12m_briar/model.pt", local_dir="./weights")

# Pre-trained Weights
hf_hub_download(repo_id="kartiknarayan/PETALface", filename="swin_arcface_webface4m/model.pt", local_dir="./weights")
hf_hub_download(repo_id="kartiknarayan/PETALface", filename="swin_cosface_webface4m/model.pt", local_dir="./weights")
hf_hub_download(repo_id="kartiknarayan/PETALface", filename="swin_arcface_webface12m/model.pt", local_dir="./weights")
hf_hub_download(repo_id="kartiknarayan/PETALface", filename="swin_cosface_webface12m/model.pt", local_dir="./weights")

Citation

@article{narayan2024petalface,
  title={PETALface: Parameter Efficient Transfer Learning for Low-resolution Face Recognition},
  author={Narayan, Kartik and Nair, Nithin Gopalakrishnan and Xu, Jennifer and Chellappa, Rama and Patel, Vishal M},
  journal={arXiv preprint arXiv:2412.07771},
  year={2024}
}

Please check our GitHub repository for complete instructions.