Cross's picture

Cross

dillfrescott

·

https://dill.moe

dillfrescott

AI & ML interests

AI, anime, computers

Recent Activity

new activity about 4 hours ago

fishaudio/fish-speech-1.5:That is not free software, as you forbid commercial use

reacted to m-ric's post with 👍 about 16 hours ago

After 6 years, BERT, the workhorse of encoder models, finally gets a replacement: 𝗪𝗲𝗹𝗰𝗼𝗺𝗲 𝗠𝗼𝗱𝗲𝗿𝗻𝗕𝗘𝗥𝗧! 🤗 We talk a lot about ✨Generative AI✨, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models. The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs). It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs. Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub. ➡️ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT. 𝗧𝗟;𝗗𝗥: 🏛️ Architecture changes: ⇒ First, standard modernizations: - Rotary positional embeddings (RoPE) - Replace GeLU with GeGLU, - Use Flash Attention 2 ✨ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead. 🥇 As a result, the model tops the game of encoder models: It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster! Read the blog post 👉 https://huggingface.co/blog/modernbert

reacted to m-ric's post with 🔥 about 16 hours ago

After 6 years, BERT, the workhorse of encoder models, finally gets a replacement: 𝗪𝗲𝗹𝗰𝗼𝗺𝗲 𝗠𝗼𝗱𝗲𝗿𝗻𝗕𝗘𝗥𝗧! 🤗 We talk a lot about ✨Generative AI✨, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models. The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs). It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs. Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub. ➡️ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT. 𝗧𝗟;𝗗𝗥: 🏛️ Architecture changes: ⇒ First, standard modernizations: - Rotary positional embeddings (RoPE) - Replace GeLU with GeGLU, - Use Flash Attention 2 ✨ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead. 🥇 As a result, the model tops the game of encoder models: It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster! Read the blog post 👉 https://huggingface.co/blog/modernbert

View all activity

Organizations

dillfrescott's activity

upvoted a collection 11 months ago

State-of-Art Markr's language model

0 items • Updated Jan 30 • 1

upvoted 3 collections 12 months ago

Mistral 7B 16k

All Mistral based models that have a 16k context size and have been finetuned. • 7 items • Updated Dec 11, 2023 • 4

Mistral 11B

Mistral models with 11B parameters. • 13 items • Updated Dec 11, 2023 • 2

Dare LLM Merges

These are large language models merged through my implementation of Super Mario DARE merge. • 10 items • Updated Dec 31, 2023 • 2

upvoted a collection about 1 year ago

Recent models: last 100 repos, sorted by creation date

The last 100 repos I have created. Sorted by creation date descending, so the most recently created repos appear at the top. • 121 items • Updated Jan 31 • 506