mpai-community/blip-vqa-base-onnx

BLIP VQA base, in ONNX

An ONNX export of Salesforce/blip-vqa-base (BLIP, visual question answering, base size), made by MPAI - Moving Picture, Audio and Data Coding by Artificial Intelligence - for Thalia, the avatar of MPAI as a Service, whose App AMQ (Multimodal Question Answering) answers questions about a picture with it. The software is at https://github.com/mpai-community/mpai-sw, in AIMs/MMC/V2.5/TIQ.

The model is not changed. The weights are Salesforce's; they are only written in another format and split in three graphs so that a program without Python can run them. This is not an official Salesforce release.

The files

The three graphs, and the weights of the first, which the first .onnx file reads from its side: keep blip_vision_model.onnx.data in the same folder.

File	Size (bytes)	SHA-256
blip_vision_model.onnx	740,384	161F6D21D7380C22568F4E41A4F764410CF157BE9FAC75E3BB064EA4DEBB82DC
blip_vision_model.onnx.data	344,457,216	1B099A7F3F04A41127DD6CD54CBBFBDEF832518F15A4A6C2CD9AEB4ABA4A5C80
blip_text_encoder_wrapper.onnx	549,391,651	BA2BACC981297823B2ADE32D49ECCA847DB4A39CC6637EB716EAD45DD5E2C530
blip_decoder_context_dynamic.onnx	645,706,276	554B87CD9892D34CBF9029F2AA4B9EE8D4513AC561EB78EC64D17646F0191991

The tokenizer vocabulary is not repeated here: it is vocab.txt of the source model (SHA-256 07ECED375CEC144D27C900241F3E339478DEC958F92FDDBC551F295C992038A3).

What each graph does

Graph	Inputs	Outputs
blip_vision_model (ONNX opset 18)	pixel_values float32 [1, 3, 384, 384]	last_hidden_state [1, 577, 768]; layer_norm_25 [1, 768]
blip_text_encoder_wrapper (opset 17)	input_ids int64 [1, n], attention_mask int64 [1, n], encoder_hidden_states float32 [1, 577, 768]	last_hidden_state [1, n, 768]; n, the length of the question, is dynamic
blip_decoder_context_dynamic (opset 17)	input_ids int64 [1, m], encoder_hidden_states float32 [1, k, 768]	logits [1, m, 30524]; m and k are dynamic

How an answer is made: the picture goes through the vision graph (resized to 384 x 384, rescaled to 0..1, normalised with the mean 0.48145466, 0.4578275, 0.40821073 and standard deviation 0.26862954, 0.26130258, 0.27577711, as the source model's preprocessor says); the question, tokenised as the source model does, and the picture's last_hidden_state go through the question encoder; and the answer is generated one token at a time by the decoder, started with the token 30522, from the encoder's output, taking the most likely token each time until the end token. AIMs/MMC/V2.5/TIQ in the repository above is a complete implementation in C#.

How it was made

With PyTorch 2.12 and the transformers library, from Salesforce/blip-vqa-base: the vision encoder (opset 18), the question encoder and the answer decoder (opset 17, the exporter without dynamo), with the question length and the context length left dynamic. The recipe is docs/conversion/export_blip_vqa_to_onnx.py in https://github.com/mpai-community/mpai-sw. It has not been re-run with the latest versions of the libraries.

Limitations

• It answers short questions about a picture with a short answer, as the source model does, and can be wrong or vague. It has the biases of the data BLIP was trained on. Do not use it where a wrong answer can hurt anyone.
• It is the base model: larger BLIP models answer better.
• Not tested on a GPU; the files are used by Thalia on the CPU.

Licence and credit

BSD 3-Clause License, Copyright (c) 2022, Salesforce.com, Inc. All rights reserved. The licence text is in LICENSE. The weights and the architecture are Salesforce's. If you use the model, cite:

@article{li2022blip,
  title     = {BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
  author    = {Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},
  journal   = {arXiv preprint arXiv:2201.12086},
  year      = {2022}
}

The conversion, and these files, are by MPAI (https://mpai.community) and are under the same licence.