malper/hierarcaps-clip-b

HierarCaps CLIP-B

Fine-tuned CLIP-B (ViT-B/32) checkpoint from the paper Emergent Visual-Semantic Hierarchies in Image-Text Representations (ECCV 2024).

• Paper: https://arxiv.org/abs/2407.08521
• Project page: https://hierarcaps.github.io/
• Code: https://github.com/TAU-VAILab/hierarcaps
• Base model: openai/clip-vit-base-patch32

This model was fine-tuned on the HierarCaps dataset to enhance hierarchical organization in CLIP's embedding space, improving hierarchical reasoning about specificity between images and texts.

Usage

from transformers import CLIPModel, AutoProcessor

model = CLIPModel.from_pretrained("malper/hierarcaps-clip-b")
processor = AutoProcessor.from_pretrained("malper/hierarcaps-clip-b")

See the official code repository for training and evaluation scripts used with this checkpoint.

Citation

@InProceedings{alper2024hierarcaps,
  author    = {Morris Alper and Hadar Averbuch-Elor},
  title     = {Emergent Visual-Semantic Hierarchies in Image-Text Representations},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2024}
}