infgrad

/

jasper_en_vision_language_v1

like 10

Hello @infgrad !

Preface

Congratulations on this release! In your space time, too, quite impressive. Looking forward to seeing people experiment with this.
I'm also curious how you used the excellent https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu to train a strong embedding model - perhaps it can be reproduced with the brand new https://huggingface.co/datasets/HuggingFaceFW/fineweb-2 for multilinguality as well?

Pull Request overview

• Introduce a custom Sentence Transformer module, based on the common Transformer module.
• Add a Normalize module as well, to always normalize (this can be useful)
• Add "padding_side": "right" to the sentence_bert_config.json. FYI: this file contains the "defaults" for the new MultiModalTransformer class, so we can just adds our desired defaults there.
• Update README example

Details

This model is a textbook example of why I recently added the possibility for custom Sentence Transformer Modules (docs). In short, we can extend any of the existing modules (or make completely new ones) and extend e.g. tokenize, forward, __init__, etc. As you can see here, this allows you to add multimodality without introducing any work to the end user.

My module is simply your custom tokenize and forward put into a module, with two small changes: the forward call is now in charge of ensuring that the pixel_values is of the correct dtype, and we use max_seq_length as the tokenizer max length instead of being hardcoded to 1024.

All that remains is explaining to the user what kind of inputs your tokenize method expects, i.e. what the user can input for correct outputs.

• Tom Aarsen