tiiuae/falcon-11B-vlm

🚀 Falcon2-11B-vlm

Falcon2-11B-vlm is an 11B parameters causal decoder-only model built by TII and trained on over 5,000B tokens of RefinedWeb enhanced with curated corpora. To bring vision capabilities, we integrate the pretrained CLIP ViT-L/14 vision encoder with our Falcon2-11B chat-finetuned model and train with image-text data. For enhancing the VLM's perception of fine-grained details w.r.t small objects in images, we employ a dynamic encoding mechanism at high-resolution for image inputs. The model is made available under the TII Falcon License 2.0, the permissive Apache 2.0-based software license which includes an acceptable use policy that promotes the responsible use of AI.

Paper coming soon 😊.

🤗 To get started with Falcon-vlm (inference, finetuning, quantization, etc.), we recommend reading this great blogpost from HF!

from transformers import LlavaNextForConditionalGeneration, LlavaNextProcessor
from PIL import Image
import requests
import torch

processor = LlavaNextProcessor.from_pretrained("tiiuae/falcon-11B-vlm", tokenizer_class='PreTrainedTokenizerFast')
model = LlavaNextForConditionalGeneration.from_pretrained("tiiuae/falcon-11B-vlm", torch_dtype=torch.bfloat16)


url = "http://images.cocodataset.org/val2017/000000039769.jpg"
cats_image = Image.open(requests.get(url, stream=True).raw)
instruction = 'Write a long paragraph about this picture.'

prompt = f"""User:<image>\n{instruction} Falcon:"""
inputs = processor(prompt, images=cats_image, return_tensors="pt", padding=True).to('cuda:0')

model.to('cuda:0')
output = model.generate(**inputs, max_new_tokens=256)


prompt_length = inputs['input_ids'].shape[1]
generated_captions = processor.decode(output[0], skip_special_tokens=True).strip()

print(generated_captions)

💥 Falcon VLMs require PyTorch 2.0 for use with transformers!

For fast inference with Falcon, check-out Text Generation Inference! Read more in this blogpost.

Model Card for Falcon2-11B-VLM

Model Details

Model Description

• Developed by: https://www.tii.ae
• Model type: Causal decoder-only
• Language(s) (NLP): English.
• License: TII Falcon License 2.0

Model Source

• Paper: coming soon.

Uses

Direct Use

Research on General large vison language models.

Out-of-Scope Use

Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful.

Bias, Risks, and Limitations

Falcon2-11B is trained mostly on English, but also German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish. It will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online.

Training Details

The training is done in two stages: pretraining and finetuning. In both stages, the visual encoder weights are kept frozen. In the pretraining stage, the LLM is kept frozen and only the multimodal projector is trained on 558K image-caption pairs. This enables the multimodal projector to learn a mapping from visual to text embedding space. During finetuning, both the projector and LLM weights are trained on a corpus of 1.2M image-text instruction data from public datasets, which also includes multi-round conversations. Falcon2-11B-VLM was trained on 16 A100 80GB GPUs with ZeRO and Flash-Attention 2.

The data was tokenized with the Falcon-7B/11B tokenizer.

Training Hyperparameters

Hyperparameter	Value
Precision	bfloat16
Optimizer	AdamW
Max learning rate	2e-5
Weight decay	0
Batch size	256

Evaluation

Model	MME	GQA	SQA	POPE	VQAv2	TextVQA	MM-Bench	SEED-IMG
Falcon2-11B VLM	1589/343	64.5	74.9	88.4	82.1	66.7	72.0	72.3

Citation

Paper coming soon 😊.

License

Falcon2-11B is licenced under TII Falcon License 2.0, the permissive Apache 2.0-based software license which includes an acceptable use policy that promotes the responsible use of AI.

Contact

falconllm@tii.ae