ColPali: Efficient Document Retrieval with Vision Language Models 👀
•
180
Manuel Faysse
manu
AI & ML interests
NLP, Privacy, multi-modal DL
Recent Activity
updated
a model
4 days ago
vidore/colpali2-3b-pt-448
updated
a model
4 days ago
vidore/colpaligemma2-3b-pt-448-base
updated
a model
18 days ago
vidore/colsmolvlm-alpha
Articles
Organizations
Posts
1
Post
These past months, I've been busy baking a special sort of Croissant 🥐 with an awesome team !
🥐 CroissantLLM is a truly bilingual language model trained on 3 trillion tokens of French and English data. In its size category (<2B), it is the best model in French, but it also rivals the best monolingual English models !
💾 To train it, we collected, filtered and cleaned huge quantities of permissively licensed French data, across various domains (legal, administrative, cultural, scientific), and different text modalities (speech transcriptions, movie subtitles, encyclopedias, forums, webpages)...
⚖️ Assessing LLM performance is not easy, especially outside of English, and to this end we crafted a novel evaluation benchmark, FrenchBench, aiming to assess reasoning, factual knowledge, and linguistic capabilities of models in French !
🔎 The best current LLMs are hidden behind a shroud of mystery, trained with undisclosed training data mixes or strategies. We go the opposite way, releasing all of the project's artefacts (model checkpoints, data, training details, evaluation benchmarks...) We obtain 81 % of the Stanford FMTI transparency criterias, far ahead of even most open initiatives !
🧪Beyond a powerful industrial resource, our transparent initiative is a stepping stone for many scientific questions ! How does teaching a model two languages instead of one splits its monolingual ability ? Does training on so much French help the model integrate French-centric knowledge and cultural biases ? How does the model memorize the training data ?
Many more things to say, for those interested, I recommend checking out:
🗞️ The blogpost: https://huggingface.co/blog/manu/croissant-llm-blog
📖 The 45 page report with lots of gems: https://arxiv.org/abs/2402.00786
🤖 Models, Data, Demo: https://huggingface.co/croissantllm
🥐 CroissantLLM is a truly bilingual language model trained on 3 trillion tokens of French and English data. In its size category (<2B), it is the best model in French, but it also rivals the best monolingual English models !
💾 To train it, we collected, filtered and cleaned huge quantities of permissively licensed French data, across various domains (legal, administrative, cultural, scientific), and different text modalities (speech transcriptions, movie subtitles, encyclopedias, forums, webpages)...
⚖️ Assessing LLM performance is not easy, especially outside of English, and to this end we crafted a novel evaluation benchmark, FrenchBench, aiming to assess reasoning, factual knowledge, and linguistic capabilities of models in French !
🔎 The best current LLMs are hidden behind a shroud of mystery, trained with undisclosed training data mixes or strategies. We go the opposite way, releasing all of the project's artefacts (model checkpoints, data, training details, evaluation benchmarks...) We obtain 81 % of the Stanford FMTI transparency criterias, far ahead of even most open initiatives !
🧪Beyond a powerful industrial resource, our transparent initiative is a stepping stone for many scientific questions ! How does teaching a model two languages instead of one splits its monolingual ability ? Does training on so much French help the model integrate French-centric knowledge and cultural biases ? How does the model memorize the training data ?
Many more things to say, for those interested, I recommend checking out:
🗞️ The blogpost: https://huggingface.co/blog/manu/croissant-llm-blog
📖 The 45 page report with lots of gems: https://arxiv.org/abs/2402.00786
🤖 Models, Data, Demo: https://huggingface.co/croissantllm
spaces
5
models
50
manu/test-opt
Updated
•
7
manu/colqwen2-7b-v0.1
Updated
•
11
manu/colqwen2-hardneg-128-5e
Updated
•
6
manu/biqwen2-warmup-0e
Updated
•
9
manu/biqwen2-warmup-1e
Updated
•
1
manu/colqwen2-v0.2
Updated
•
2
manu/biqwen2-1e-256-newpad-nobuffer
Updated
•
1
manu/colqwen2-ba64
Updated
•
2
manu/dse_1024
Updated
manu/marco_mcdse
Updated
datasets
69
manu/colpali-queries
Viewer
•
Updated
•
118k
•
46
•
2
manu/colpali-corpus
Viewer
•
Updated
•
39.2k
•
580
•
1
manu/canary_split
Updated
•
8
manu/wnut_17
Viewer
•
Updated
•
5.69k
•
52
manu/colpali-data-ir
Viewer
•
Updated
•
118k
•
42
•
1
manu/tok_dataset_1_newtok
Viewer
•
Updated
•
9.73k
•
36
manu/tok_dataset_1
Viewer
•
Updated
•
9.75k
•
39
manu/dataset_en_fr_short
Viewer
•
Updated
•
1.27k
•
39
manu/dataset_en_fr
Viewer
•
Updated
•
2.46k
•
38
manu/dataset_1
Viewer
•
Updated
•
9.76k
•
45