s k's picture

1 5 9

s k

madstuntman11

·

AI & ML interests

None yet

Recent Activity

Reacted to merve's post with ❤️ about 1 month ago

If you have documents that do not only have text and you're doing retrieval or RAG (using OCR and LLMs), give it up and give ColPali and vision language models a try 🤗 Why? Documents consist of multiple modalities: layout, table, text, chart, images. Document processing pipelines often consist of multiple models and they're immensely brittle and slow. 🥲 How? ColPali is a ColBERT-like document retrieval model built on PaliGemma, it operates over image patches directly, and indexing takes far less time with more accuracy. You can use it for retrieval, and if you want to do retrieval augmented generation, find the closest document, and do not process it, give it directly to a VLM like Qwen2-VL (as image input) and give your text query. 🤝 This is much faster + you do not lose out on any information + much easier to maintain too! 🥳 Multimodal RAG https://huggingface.co/collections/merve/multimodal-rag-66d97602e781122aae0a5139 💬 Document AI (made it way before, for folks who want structured input/output and can fine-tune a model) https://huggingface.co/collections/merve/awesome-document-ai-65ef1cdc2e97ef9cc85c898e 📖

Reacted to merve's post with ❤️ about 1 month ago

I have put together a notebook on Multimodal RAG, where we do not process the documents with hefty pipelines but natively use: - https://huggingface.co/vidore/colpali for retrieval 📖 it doesn't need indexing with image-text pairs but just images! - https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct for generation 💬 directly feed images as is to a vision language model with no processing to text! I used ColPali implementation of the new 🐭 Byaldi library by @bclavie 🤗 https://github.com/answerdotai/byaldi Link to notebook: https://github.com/merveenoyan/smol-vision/blob/main/ColPali_%2B_Qwen2_VL.ipynb

upvoted an article about 1 month ago

Document Similarity Search with ColPali

View all activity

Organizations

None yet

madstuntman11's activity

New activity in Tevatron/dse-phi3-docmatix-v1 4 months ago

Rename Tevatron-DSE-Phi3-Docmatix-V1(ZeroShot)_metrics.json to results.json

#1 opened 4 months ago by