s k's picture

1 5 9

s k

madstuntman11

·

AI & ML interests

None yet

Recent Activity

Reacted to merve's post with ❤️ about 1 month ago

If you have documents that do not only have text and you're doing retrieval or RAG (using OCR and LLMs), give it up and give ColPali and vision language models a try 🤗 Why? Documents consist of multiple modalities: layout, table, text, chart, images. Document processing pipelines often consist of multiple models and they're immensely brittle and slow. 🥲 How? ColPali is a ColBERT-like document retrieval model built on PaliGemma, it operates over image patches directly, and indexing takes far less time with more accuracy. You can use it for retrieval, and if you want to do retrieval augmented generation, find the closest document, and do not process it, give it directly to a VLM like Qwen2-VL (as image input) and give your text query. 🤝 This is much faster + you do not lose out on any information + much easier to maintain too! 🥳 Multimodal RAG https://huggingface.co/collections/merve/multimodal-rag-66d97602e781122aae0a5139 💬 Document AI (made it way before, for folks who want structured input/output and can fine-tune a model) https://huggingface.co/collections/merve/awesome-document-ai-65ef1cdc2e97ef9cc85c898e 📖

Reacted to merve's post with ❤️ about 1 month ago

I have put together a notebook on Multimodal RAG, where we do not process the documents with hefty pipelines but natively use: - https://huggingface.co/vidore/colpali for retrieval 📖 it doesn't need indexing with image-text pairs but just images! - https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct for generation 💬 directly feed images as is to a vision language model with no processing to text! I used ColPali implementation of the new 🐭 Byaldi library by @bclavie 🤗 https://github.com/answerdotai/byaldi Link to notebook: https://github.com/merveenoyan/smol-vision/blob/main/ColPali_%2B_Qwen2_VL.ipynb

upvoted an article about 1 month ago

Document Similarity Search with ColPali

View all activity

Organizations

None yet

madstuntman11's activity

liked a Space about 2 months ago

Diffusers Image Outpaint

liked a model 4 months ago

manu/colpali-3b-mix-448-docmatix-only-mined-ib

Updated Jul 31 • 3 • 2

liked a Space 4 months ago

Vidore Leaderboard

liked a model 4 months ago

nvidia/MambaVision-T-1K

Image Feature Extraction • Updated Jul 25 • 5.37k • 26

liked a Space 4 months ago

Running on Zero

Florence 2

liked a dataset 4 months ago

Tevatron/docmatix-ir

Viewer • Updated Aug 12 • 5.61M • 5.08k • 12

liked 2 models 4 months ago

Tevatron/dse-phi3-docmatix-v1

Updated Aug 12 • 55 • 9

manu/colpali-3b-mix-448-docmatix

Updated Jul 23 • 7

liked a Space 5 months ago

Chatbot Arena Leaderboard