openflamingo/OpenFlamingo-3B-vitl-mpt1b-langinstruct

OpenFlamingo-3B (CLIP ViT-L/14, MPT-1B-Dolly)

Paper | Blog post | Code | Demo

OpenFlamingo is an open source implementation of DeepMind's Flamingo models. This 3B-parameter model uses a CLIP ViT-L/14 vision encoder and an instruction-tuned MPT-1B language model.

Model Details

We follow the Flamingo modeling paradigm, outfitting the layers of a pretrained, frozen language model such that they cross-attend to visual features when decoding. Following Flamingo, we freeze the vision encoder and language model but train the connecting modules on web-scraped image-text sequences. Specifically, we trained this model on a mixture of LAION-2B and Multimodal C4.

This model has cross-attention modules inserted in every decoder block. It was trained using DistributedDataParallel across 64 A100 40GB GPUs at FP32 precision.

The MPT-1B modeling code does not accept the labels kwarg and compute cross-entropy loss within forward(). To train with the OpenFlamingo codebase, we suggest using a version with the labels kwarg here.

Uses

OpenFlamingo models process arbitrarily interleaved sequences of images and text to output text. This allows the models to accept in-context examples and undertake tasks like captioning, visual question answering, and image classification.

Initialization

from open_flamingo import create_model_and_transforms

model, image_processor, tokenizer = create_model_and_transforms(
    clip_vision_encoder_path="ViT-L-14",
    clip_vision_encoder_pretrained="openai",
    lang_encoder_path="anas-awadalla/mpt-1b-redpajama-200b-dolly",
    tokenizer_path="anas-awadalla/mpt-1b-redpajama-200b-dolly",
    cross_attn_every_n_layers=1
)

# grab model checkpoint from huggingface hub
from huggingface_hub import hf_hub_download
import torch

checkpoint_path = hf_hub_download("openflamingo/OpenFlamingo-3B-vitl-mpt1b-langinstruct", "checkpoint.pt")
model.load_state_dict(torch.load(checkpoint_path), strict=False)

Generation example

Below is an example of generating text conditioned on interleaved images/text. In particular, let's try few-shot image captioning.

from PIL import Image
import requests

"""
Step 1: Load images
"""
demo_image_one = Image.open(
    requests.get(
        "http://images.cocodataset.org/val2017/000000039769.jpg", stream=True
    ).raw
)

demo_image_two = Image.open(
    requests.get(
        "http://images.cocodataset.org/test-stuff2017/000000028137.jpg",
        stream=True
    ).raw
)

query_image = Image.open(
    requests.get(
        "http://images.cocodataset.org/test-stuff2017/000000028352.jpg", 
        stream=True
    ).raw
)


"""
Step 2: Preprocessing images
Details: For OpenFlamingo, we expect the image to be a torch tensor of shape 
 batch_size x num_media x num_frames x channels x height x width. 
 In this case batch_size = 1, num_media = 3, num_frames = 1,
 channels = 3, height = 224, width = 224.
"""
vision_x = [image_processor(demo_image_one).unsqueeze(0), image_processor(demo_image_two).unsqueeze(0), image_processor(query_image).unsqueeze(0)]
vision_x = torch.cat(vision_x, dim=0)
vision_x = vision_x.unsqueeze(1).unsqueeze(0)

"""
Step 3: Preprocessing text
Details: In the text we expect an <image> special token to indicate where an image is.
 We also expect an <|endofchunk|> special token to indicate the end of the text 
 portion associated with an image.
"""
tokenizer.padding_side = "left" # For generation padding tokens should be on the left
lang_x = tokenizer(
    ["<image>An image of two cats.<|endofchunk|><image>An image of a bathroom sink.<|endofchunk|><image>An image of"],
    return_tensors="pt",
)


"""
Step 4: Generate text
"""
generated_text = model.generate(
    vision_x=vision_x,
    lang_x=lang_x["input_ids"],
    attention_mask=lang_x["attention_mask"],
    max_new_tokens=20,
    num_beams=3,
)

print("Generated text: ", tokenizer.decode(generated_text[0]))

Bias, Risks, and Limitations

OpenFlamingo models inherit the risks of their parent models, especially the language model. As an open-source research effort, we highly value open, accessible, reproducible multimodal model research; however, it is crucial to be aware that these models are trained on web data, have not been finetuned for safety, and thus may produce unintended, inappropriate, unreliable, and/or inaccurate outputs. Please use caution before deploying OpenFlamingo models in real applications. We also hope that OpenFlamingo enables further safety and reliability research to address these issues.

In an effort to mitigate current potential biases and harms, we have deployed a text content filter on model outputs in the OpenFlamingo demo. We continue to red-team the model to understand and improve its safety.

Evaluation

	0-shot	4-shot	8-shot	16-shot	32-shot
COCO (CIDEr)	74.4 (0.6)	82.7 (0.7)	87.8 (0.5)	91.9 (0.3)	94.8 (0.3)
VQAv2 (Accuracy)	44.8 (0.7)	46.8 (0.5)	46.9 (0.9)	46.8 (0.7)	46.5 (0.5)
Flickr-30K (CIDEr)	51.2 (0.2)	59.1 (0.3)	60.7 (0.6)	63.0 (0.4)	64.5 (1.3)
OK-VQA (Accuracy)	26.2 (0.3)	31.9 (0.2)	31.4 (0.4)	31.6 (0.3)	31.0 (0.1)
TextVQA (Accuracy)	23.1 (0.2)	28.1 (0.4)	29.1 (0.1)	29.1 (0.1)	28.5 (0.1)
Vizwiz (Accuracy)	18.0 (0.6)	22.0 (0.8)	28.8 (1.3)	35.5 (0.8)	41.3 (0.5)
Hateful Memes (ROC AUC)	54.3 (2.5)	53.5 (1.1)	52.1 (2.6)	52.3 (3.0)	51.0 (2.3)