Expert Support case study: Bolstering a RAG app with LLM-as-a-Judge
•
25
Aymeric Roucher
m-ric
AI & ML interests
Leading Agents at Hugging Face 🤗
Recent Activity
liked
a Space
3 days ago
data-agents/jupyter-agent
posted
an
update
3 days ago
After 6 years, BERT, the workhorse of encoder models, finally gets a replacement: 𝗪𝗲𝗹𝗰𝗼𝗺𝗲 𝗠𝗼𝗱𝗲𝗿𝗻𝗕𝗘𝗥𝗧! 🤗
We talk a lot about ✨Generative AI✨, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models.
The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs).
It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs.
Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub.
➡️ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT.
𝗧𝗟;𝗗𝗥:
🏛️ Architecture changes:
⇒ First, standard modernizations:
- Rotary positional embeddings (RoPE)
- Replace GeLU with GeGLU,
- Use Flash Attention 2
✨ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead.
🥇 As a result, the model tops the game of encoder models:
It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster!
Read the blog post 👉 https://huggingface.co/blog/modernbert
updated
a Space
3 days ago
m-ric/ai-travel-planner
Articles
Organizations
m-ric's activity
liked
a
Space
3 days ago
liked
a
Space
13 days ago
liked
3
models
13 days ago
liked
a
Space
15 days ago
liked
a
Space
18 days ago
liked
3
Spaces
19 days ago
liked
2
datasets
20 days ago
liked
a
Space
21 days ago
liked
a
Space
23 days ago
liked
a
Space
25 days ago
liked
a
dataset
26 days ago
liked
3
Spaces
27 days ago
liked
a
Space
about 1 month ago