|
--- |
|
license: apache-2.0 |
|
tags: |
|
- vision |
|
datasets: |
|
- imagenet-1k |
|
--- |
|
|
|
# Vision Transformer (base sized model) pre-trained with MSN (patch size of 4) |
|
|
|
Vision Transformer (ViT) model pre-trained using the MSN method. It was introduced in the paper [Masked Siamese Networks for Label-Efficient Learning](https://arxiv.org/abs/2204.07141) by Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas and first released in [this repository](https://github.com/facebookresearch/msn). |
|
|
|
Disclaimer: The team releasing MSN did not write a model card for this model so this model card has been written by the Hugging Face team. |
|
|
|
## Model description |
|
|
|
The Vision Transformer (ViT) is a transformer encoder model (BERT-like). Images are presented to the model as a sequence of fixed-size patches. |
|
|
|
MSN presents a joint-embedding architecture to match the prototypes of masked patches with that of the unmasked patches. With this setup, their method yields excellent performance in the low-shot and extreme low-shot regimes. |
|
|
|
By pre-training the model, it learns an inner representation of images that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled images for instance, you can train a standard classifier by placing a linear layer on top of the pre-trained encoder. |
|
|
|
## Intended uses & limitations |
|
|
|
You can use the raw model for downstream tasks like image classification. See the [model hub](https://huggingface.co/models?filter=vit_msn) to look for different versions of MSN pre-trained models that interest you. The model is particularly beneficial when you have a few labeled samples in your training set. |
|
|
|
### How to use |
|
|
|
Here is how to use this backbone encoder: |
|
|
|
```python |
|
from transformers import AutoFeatureExtractor, ViTMSNModel |
|
import torch |
|
from PIL import Image |
|
import requests |
|
|
|
url = "http://images.cocodataset.org/val2017/000000039769.jpg" |
|
image = Image.open(requests.get(url, stream=True).raw) |
|
|
|
feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/vit-msn-base-4") |
|
model = ViTMSNModel.from_pretrained("facebook/vit-msn-base-4") |
|
inputs = feature_extractor(images=image, return_tensors="pt") |
|
with torch.no_grad(): |
|
outputs = model(**inputs) |
|
last_hidden_states = outputs.last_hidden_state |
|
``` |
|
|
|
For fine-tuning on image classification use the `ViTMSNForImageClassification` class: |
|
|
|
```python |
|
from transformers import AutoFeatureExtractor, ViTMSNForImageClassification |
|
import torch |
|
from PIL import Image |
|
import requests |
|
|
|
url = "http://images.cocodataset.org/val2017/000000039769.jpg" |
|
image = Image.open(requests.get(url, stream=True).raw) |
|
|
|
feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/vit-msn-base-4") |
|
model = ViTMSNForImageClassification.from_pretrained("facebook/vit-msn-base-4") |
|
|
|
... |
|
``` |
|
|
|
|
|
### Citation |
|
|
|
```bibtex |
|
@article{assran2022masked, |
|
title={Masked Siamese Networks for Label-Efficient Learning}, |
|
author={Assran, Mahmoud, and Caron, Mathilde, and Misra, Ishan, and Bojanowski, Piotr, and Bordes, Florian and Vincent, Pascal, and Joulin, Armand, and Rabbat, Michael, and Ballas, Nicolas}, |
|
journal={arXiv preprint arXiv:2204.07141}, |
|
year={2022} |
|
} |
|
``` |
