Integrate with Sentence Transformers v5.4
Hello!
Pull Request overview
- Integrate with Sentence Transformers v5.4
Details
This is a companion PR to https://huggingface.co/nomic-ai/nomic-embed-multimodal-3b/discussions/4, which has a lot more details.
Like that PR, this PR matches the original colpali behaviour from when the model was trained, while colpali itself regressed. The BiQwen2_5 baseline on current colpali-engine (0.3.15) shifts further to tensor([[0.1074, 0.0981], [0.0957, 0.0757]]) because process_queries was refactored to drop the "Query: " prefix and use a 10-token augmentation buffer instead of 1. The ST integration deliberately targets the training-era format, not the refactored one.
One caveat: loading a LoRA PEFT adapter on qwen2_5_vl hits a regression in transformers>=5.0.0 where _convert_peft_config_moe raises KeyError: 'qwen2_vl'. A one-line fix is on its way upstream in https://github.com/huggingface/transformers/pull/45428; until it lands, users on v5.x need to either cherry-pick that fix or stay on transformers<5 for the Sentence Transformers path. The colpali path is unaffected since it keeps its own transformers pin.
Note that none of the old behaviour is affected or changed. This PR only adds an additional way to run this model in a familiar and common format.
Let me know if you have any questions or feedback!
- Tom Aarsen