PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation

PRIMA answers questions about multiple images and grounds its explanations in segmentation masks. It compares objects and parts across images through functional, spatial, numerical, and open-ended reasoning.

Code and evaluator · Paper · Project

Model Details

• Model name: PRIMA
• Language backbone: Vicuna-7B-based decoder, initialized from MBZUAI/GLaMM-FullScope, revision 95a4ab0c9ab7e8e583df3e8adc93b5c89586612e. The inherited language weights include that checkpoint’s prior adaptation.
• Training dataset: M⁴Seg
• Task: multi-image pixel-grounded reasoning
• Checkpoint format: complete BF16 Hugging Face checkpoint in safetensors shards, with LoRA updates merged into the language-model weights
• Visual encoder: EVA-CLIP-G with an instruction-guided Q-Former
• Cross-image reasoning: SQuARE relational queries and attention
• Grounding decoder: SAM ViT-H with text-to-mask projection

The checkpoint includes the language model, visual encoder, Q-Former, SQuARE, and grounding decoder. Both the language-model tokenizer and the Q-Former tokenizer are included. No separate base-model download, LoRA adapter, or vision checkpoint is required.

Intended Use

PRIMA supports research on reasoning across images with object- and part-level grounding. Given an ordered image group and a question, it returns a natural-language response and masks grouped by source image.

Quick Start

Use the PRIMA environment: Python 3.10, PyTorch 1.13.1 with CUDA 11.7, and Transformers 4.42.3. Install the matching CUDA build first, then the inference dependencies included in this repository:

pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 \
  --extra-index-url https://download.pytorch.org/whl/cu117
pip install -r requirements.txt

Load directly from Hugging Face:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BertTokenizer

model_id = "kanguyen-vn/PRIMA"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
qformer_tokenizer = BertTokenizer.from_pretrained(
    model_id, subfolder="qformer_tokenizer", truncation_side="left"
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
).to(device="cuda", dtype=torch.bfloat16).eval()

result = model.predict(
    images=["/path/to/first_image.jpg", "/path/to/second_image.jpg"],
    question="Identify and segment the objects shared by these images.",
    tokenizer=tokenizer,
    qformer_tokenizer=qformer_tokenizer,
    max_new_tokens=1024,
)
print(result["response"])
for image_index, masks in enumerate(result["masks"]):
    if masks is not None:
        # Boolean tensor: [number of masks, original image height, original image width].
        masks = masks.cpu().numpy()

The model uses custom code included with the checkpoint, so trust_remote_code=True is required. Use predict for image-grounded responses; a generic text-generation or image-segmentation pipeline does not prepare PRIMA's multi-image inputs. The validated inference path uses BF16 on one CUDA GPU. Tokenizers and weights can also be loaded from a downloaded local snapshot without network access.

Evaluation

The complete inference, training, and evaluation source is included under code/. The generation driver, generation implementation, metric implementation, and evaluation guide are directly available here.

The semantic metric model is sentence-transformers/all-MiniLM-L6-v2, matching the paper's metric implementation. Normalized sentence embeddings are compared by cosine similarity for grounded labels and cleaned answers. This release pins the metric encoder to revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. This metric encoder is separate from PRIMA's language backbone. METEOR uses PTB-tokenized COCO caption evaluation and requires Java.

Download the source and install its dependencies in the PRIMA environment described above:

huggingface-cli download kanguyen-vn/PRIMA \
  --include "code/*" --local-dir ./prima_release
cd prima_release/code
pip install -r requirements.txt
huggingface-cli download kanguyen-vn/M4Seg \
  --repo-type dataset --local-dir ./data/m4seg \
  --include "annotations/*" "metadata/*" "prepare_images.py" "PREPARATION.md"

Obtain source images and create the layout described in the M⁴Seg preparation guide. With the prepared image root at ./data, generate and score the original test split using:

python evaluate.py \
  --merged_ckpt_path kanguyen-vn/PRIMA \
  --dataset_dir ./data \
  --m4seg_annpath ./data/m4seg/annotations/m4seg_test.json.gz \
  --output_save_path ./outputs/m4seg_test \
  --compute_metrics \
  --text_model sentence-transformers/all-MiniLM-L6-v2

To score saved predictions separately:

python -m eval.m4seg ./outputs/m4seg_test \
  --gt-json ./data/m4seg/annotations/m4seg_test.json.gz \
  --penalize-missing \
  --with-text --text-model sentence-transformers/all-MiniLM-L6-v2 --with-meteor \
  --output ./outputs/m4seg_test/metrics.json

The report includes mIoU, Recall, I-SS, I-SIoU, SS, SIoU, and METEOR. Missing or failed predictions receive zero for mask and phrase scores under this command. METEOR uses annotated images with predicted masks, with coverage recorded as meteor_evaluated_images. Grounding masks are matched greedily within each annotated image; unmatched masks are excluded from the matched-pair mIoU denominator. The metric report records the semantic encoder and its resolved revision.

Evaluation can resume with --resume when the saved model, annotations, generation settings, preprocessing, source code, and dependencies match. Each shard needs its own output directory. The scorer rejects duplicate example IDs across shards. Older predictions without matching provenance can still be scored separately.

Training and Fine-Tuning

PRIMA was fine-tuned on M⁴Seg using language-model LoRA, the SQuARE query and attention parameters, the visual projector, the text-to-mask projection, and the SAM mask decoder. This inference checkpoint contains the merged weights.

License

Original PRIMA code and model contributions are licensed under MIT, with inherited pretrained-component terms retained. See LICENSE.md for the scope and full MIT text, and LICENSES for inherited code licenses.

Citation

@inproceedings{nguyen2027prima,
  title     = {PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation},
  author    = {Nguyen, Kiet A. and Wahed, Muntasir and Juvekar, Adheesh Sunil and
               Li, Xinzhuo and Zhou, Xiaona and Shah, Vedant and Yu, Tianjiao and
               Yanardag, Pinar and Lourentzou, Ismini},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  year      = {2027}
}

Acknowledgments and Third-party Notices

PRIMA builds on GLaMM, LLaVA, Segment Anything, EVA, and LAVIS/BLIP. The released model combines these pretrained components with PRIMA's trained parameters. Original copyright and license headers are retained in the accompanying source; upstream model and component terms continue to apply.