Open-source LLMs as LangChain Agents
โข
6
Aymeric Roucher
m-ric
AI & ML interests
MLE at Hugging Face ๐ค
LLMs, Agents, RAG, Multimodal.
Articles
Organizations
m-ric's activity
posted
an
update
10 days ago
Post
2509
๐ฐโ ๐๐๐ฌ๐๐๐ซ๐๐ก ๐๐จ๐ซ ๐ญ๐ก๐ ๐ฏ๐๐ซ๐ฒ ๐๐๐ ๐๐จ๐จ๐ซ - ๐๐๐๐ฅ๐ข๐ง๐ ๐ฅ๐๐ฐ๐ฌ ๐ซ๐๐ฉ๐ฅ๐ข๐๐๐ญ๐ข๐จ๐ง
๐ Good news: ๐๐ผ๐ ๐ฐ๐ฎ๐ป ๐ฑ๐ผ ๐ฐ๐๐๐๐ถ๐ป๐ด-๐ฒ๐ฑ๐ด๐ฒ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐๐ถ๐๐ต ๐ฎ ๐ฐ๐ฎ๐น๐ฐ๐๐น๐ฎ๐๐ผ๐ฟ ๐ฎ๐ป๐ฑ ๐ ๐ถ๐ฐ๐ฟ๐ผ๐๐ผ๐ณ๐ ๐ฃ๐ฎ๐ถ๐ป๐ ๐ฎ๐ฌ๐ฌ๐ฒ!
The Chinchilla experiments (by Google DeepMind) ran hundreds of pre-trainings with models >1B parameters (I do not want to imagine how much that cost) to ๐ณ๐ถ๐ป๐ฑ ๐๐ต๐ฒ ๐ผ๐ฝ๐๐ถ๐บ๐ฎ๐น ๐ฟ๐ฎ๐๐ถ๐ผ ๐ผ๐ณ ๐บ๐ผ๐ฑ๐ฒ๐น ๐๐ถ๐๐ฒ ๐๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐๐ผ๐ธ๐ฒ๐ป๐. Why is this question so important?
Well, you only ever have access to a fixed compute, counted in FLOPs (floating point operations). So if your model is bigger, you will have less compute to train on many tokens, and if you want to train on more tokens, your model will be smaller. When model trainings cost million, you absolutely need to get this right.
The new paper "Chinchilla Scaling: A replication attempt" by Epoch AI sets on on the ambitious goal of reproducing this.
But since the authors do not have infinite money, they decided to directly run their computations from DeepMind's own experiments! They took the figure from the last experiment (cf slide below), measured point positions, picked color codes, and ended up reconstructing the underlying data.
๐ฅ They then just fit the scaling laws proposed by the Chinchilla Authors, but arrived at wildly different results! They find that as a rough rule of thumb, you should use 20 training tokens for each parameter in your model, instead of the 70 obtained in the original paper. They also point out inconsistencies in the paper, and unrealistically narrow confidence intervals.
โก๏ธ This only contradicts the results from the last (out of 3) experiments in the Chinchilla paper. And the model trained at the end of the Chinchilla paper still seems properly scaled.
โ But it does show that a tiny bit more theoretical work can go a long way, especially given the huge financial costs that such an error can have!
๐ Good news: ๐๐ผ๐ ๐ฐ๐ฎ๐ป ๐ฑ๐ผ ๐ฐ๐๐๐๐ถ๐ป๐ด-๐ฒ๐ฑ๐ด๐ฒ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐๐ถ๐๐ต ๐ฎ ๐ฐ๐ฎ๐น๐ฐ๐๐น๐ฎ๐๐ผ๐ฟ ๐ฎ๐ป๐ฑ ๐ ๐ถ๐ฐ๐ฟ๐ผ๐๐ผ๐ณ๐ ๐ฃ๐ฎ๐ถ๐ป๐ ๐ฎ๐ฌ๐ฌ๐ฒ!
The Chinchilla experiments (by Google DeepMind) ran hundreds of pre-trainings with models >1B parameters (I do not want to imagine how much that cost) to ๐ณ๐ถ๐ป๐ฑ ๐๐ต๐ฒ ๐ผ๐ฝ๐๐ถ๐บ๐ฎ๐น ๐ฟ๐ฎ๐๐ถ๐ผ ๐ผ๐ณ ๐บ๐ผ๐ฑ๐ฒ๐น ๐๐ถ๐๐ฒ ๐๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐๐ผ๐ธ๐ฒ๐ป๐. Why is this question so important?
Well, you only ever have access to a fixed compute, counted in FLOPs (floating point operations). So if your model is bigger, you will have less compute to train on many tokens, and if you want to train on more tokens, your model will be smaller. When model trainings cost million, you absolutely need to get this right.
The new paper "Chinchilla Scaling: A replication attempt" by Epoch AI sets on on the ambitious goal of reproducing this.
But since the authors do not have infinite money, they decided to directly run their computations from DeepMind's own experiments! They took the figure from the last experiment (cf slide below), measured point positions, picked color codes, and ended up reconstructing the underlying data.
๐ฅ They then just fit the scaling laws proposed by the Chinchilla Authors, but arrived at wildly different results! They find that as a rough rule of thumb, you should use 20 training tokens for each parameter in your model, instead of the 70 obtained in the original paper. They also point out inconsistencies in the paper, and unrealistically narrow confidence intervals.
โก๏ธ This only contradicts the results from the last (out of 3) experiments in the Chinchilla paper. And the model trained at the end of the Chinchilla paper still seems properly scaled.
โ But it does show that a tiny bit more theoretical work can go a long way, especially given the huge financial costs that such an error can have!
posted
an
update
24 days ago
Post
2290
๐๐๐ฉ๐๐ซ ๐๐๐ฏ๐ข๐๐ฐ: ๐๐ก๐จ-๐ - ๐๐จ ๐ง๐จ๐ญ ๐ฎ๐ฌ๐ ๐๐ฅ๐ฅ ๐ญ๐จ๐ค๐๐ง๐ฌ ๐๐ช๐ฎ๐๐ฅ๐ฅ๐ฒ ๐ข๐ง ๐ฒ๐จ๐ฎ๐ซ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ! โ๏ธโ๏ธ
A new paper topping Daily papers questions a hidden assumption in LLM training:
๐ค ๐๐๐ค๐ช๐ก๐ ๐ฌ๐ ๐ง๐๐๐ก๐ก๐ฎ ๐ช๐จ๐ ๐๐ก๐ก ๐ฉ๐ค๐ ๐๐ฃ๐จ ๐๐ฆ๐ช๐๐ก๐ก๐ฎ ๐๐ฃ ๐ค๐ช๐ง ๐๐๐'๐จ ๐ฉ๐ง๐๐๐ฃ๐๐ฃ๐ ?
Some tokens are more relevant than others, and some are mostly noise (just look up the history of ๐๐ฐ๐ญ๐ช๐ฅ๐๐ฐ๐ญ๐ฅ๐๐ข๐จ๐ช๐ฌ๐ข๐ณ๐ฑ).
So this paper introduces ๐ฆ๐ฒ๐น๐ฒ๐ฐ๐๐ถ๐๐ฒ ๐๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐ ๐ผ๐ฑ๐ฒ๐น๐ถ๐ป๐ด, which is actually really simple:
โก๏ธ A specific metric measures the relevance of each token. Then during training, only the top k% tokens for this relevance metric count in the loss calculation.
Authors test this method by training models on the difficult MATH dataset (only competition mathematics problems).
โก๏ธ Their technique seems like a new must-do in LLM training: Training is much faster and reaches an impressive performance!
๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
โ โฑ๏ธ Training is x5 to x10 faster to reach equivalent performance compared to standard language modeling.
โ ๐ช Their 1B model achieves close to GPT4 Chain-of-Thought performance on MATH!
โ ๐ Their 7B model match performance of the state-of-the-art DeepSeek for the same size, while trained on only 3% of tokens
๐๐๐๐ข๐ญ๐ข๐จ๐ง๐๐ฅ ๐ข๐ง๐ฌ๐ข๐ ๐ก๐ญ๐ฌ ๐ก
โ Datasets used for pre-training, even after pre-filtering, still contain a large proportion of noisy tokens ๐
โ Authors show that when you reduce loss on noisy tokens, you actually reduce accuracy (Figure 7). So Selective Language Modeling seems fundamental! โ
Find great reads in @akhaliq 's Daily Papers ๐ https://huggingface.co/papers
Paper added to my collection ๐ m-ric/spinning-up-in-llms-659e698f9dd5a71bd3f579a7
A new paper topping Daily papers questions a hidden assumption in LLM training:
๐ค ๐๐๐ค๐ช๐ก๐ ๐ฌ๐ ๐ง๐๐๐ก๐ก๐ฎ ๐ช๐จ๐ ๐๐ก๐ก ๐ฉ๐ค๐ ๐๐ฃ๐จ ๐๐ฆ๐ช๐๐ก๐ก๐ฎ ๐๐ฃ ๐ค๐ช๐ง ๐๐๐'๐จ ๐ฉ๐ง๐๐๐ฃ๐๐ฃ๐ ?
Some tokens are more relevant than others, and some are mostly noise (just look up the history of ๐๐ฐ๐ญ๐ช๐ฅ๐๐ฐ๐ญ๐ฅ๐๐ข๐จ๐ช๐ฌ๐ข๐ณ๐ฑ).
So this paper introduces ๐ฆ๐ฒ๐น๐ฒ๐ฐ๐๐ถ๐๐ฒ ๐๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐ ๐ผ๐ฑ๐ฒ๐น๐ถ๐ป๐ด, which is actually really simple:
โก๏ธ A specific metric measures the relevance of each token. Then during training, only the top k% tokens for this relevance metric count in the loss calculation.
Authors test this method by training models on the difficult MATH dataset (only competition mathematics problems).
โก๏ธ Their technique seems like a new must-do in LLM training: Training is much faster and reaches an impressive performance!
๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
โ โฑ๏ธ Training is x5 to x10 faster to reach equivalent performance compared to standard language modeling.
โ ๐ช Their 1B model achieves close to GPT4 Chain-of-Thought performance on MATH!
โ ๐ Their 7B model match performance of the state-of-the-art DeepSeek for the same size, while trained on only 3% of tokens
๐๐๐๐ข๐ญ๐ข๐จ๐ง๐๐ฅ ๐ข๐ง๐ฌ๐ข๐ ๐ก๐ญ๐ฌ ๐ก
โ Datasets used for pre-training, even after pre-filtering, still contain a large proportion of noisy tokens ๐
โ Authors show that when you reduce loss on noisy tokens, you actually reduce accuracy (Figure 7). So Selective Language Modeling seems fundamental! โ
Find great reads in @akhaliq 's Daily Papers ๐ https://huggingface.co/papers
Paper added to my collection ๐ m-ric/spinning-up-in-llms-659e698f9dd5a71bd3f579a7
posted
an
update
30 days ago
Post
2067
๐ก๐ฒ๐ ๐ฆ๐ฝ๐ฎ๐ฐ๐ฒ: ๐ผ๐ ๐๐ง๐๐ซ๐๐ก ๐ฅ๐ก๐๐ฃ๐ฃ๐๐ง ๐บ๏ธ๐๏ธ Plan your next vacation in a few minutes!
I wanted to try out if a powerful LLM like Mixtral-8x7b had geographical reasoning capabilities.
So I built a small space that prompts the LLM to provide a JSON list of places based on a user input.
And the result was impressive! ๐คฏ
โ ๐๐ ๐๐ฒ๐ฒ๐บ๐ ๐น๐ถ๐ธ๐ฒ ๐ ๐ถ๐ ๐๐ฟ๐ฎ๐น ๐ต๐ฎ๐ ๐ฎ ๐ด๐ฟ๐ฎ๐๐ฝ ๐ผ๐ณ ๐ด๐ฒ๐ผ๐ด๐ฟ๐ฎ๐ฝ๐ต๐ถ๐ฐ๐ฎ๐น ๐ฐ๐ผ๐ป๐ฐ๐ฒ๐ฝ๐๐ ๐น๐ถ๐ธ๐ฒ ๐ก๐ผ๐ฟ๐๐ต - ๐ฆ๐ผ๐๐๐ต, ๐ผ๐ฟ ๐๐ฝ๐ฎ๐๐ถ๐ฎ๐น ๐ฎ๐น๐ถ๐ด๐ป๐บ๐ฒ๐ป๐.๐งญ Not just describing these concepts, but really applying them in practice, for instance to successfully answer "give me 4 European cities that are aligned on the map". This is a ๐ป๐ถ๐ฐ๐ฒ ๐ฒ๐ ๐ฎ๐บ๐ฝ๐น๐ฒ ๐ผ๐ณ ๐ฎ๐ป ๐ฒ๐บ๐ฒ๐ฟ๐ด๐ฒ๐ป๐ ๐ฐ๐ฎ๐ฝ๐ฎ๐ฏ๐ถ๐น๐ถ๐๐, since nothing in the LLM's training data should prepare it for this specific task.
Anyway, I added API calls and a nice visualization on top of the LLM, streaming output, caching for the answers and locations... and ta-da! โจ I got the ๐๐ ๐ง๐ฟ๐ฎ๐๐ฒ๐น ๐ฃ๐น๐ฎ๐ป๐ป๐ฒ๐ฟ.
๐๐ค๐ช ๐๐๐ฃ ๐๐๐จ๐๐ง๐๐๐ ๐๐ฉ ๐ฎ๐ค๐ช๐ง ๐ฉ๐ง๐๐ฅ, ๐๐ฃ๐ ๐๐ฉ ๐ฌ๐๐ก๐ก ๐๐ค๐ข๐ ๐ช๐ฅ ๐ฌ๐๐ฉ๐ ๐ฃ๐๐๐ ๐๐ฃ๐ ๐๐ค๐ฃ๐ซ๐๐ฃ๐๐๐ฃ๐ฉ ๐ก๐ค๐๐๐ฉ๐๐ค๐ฃ๐จ!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐ ๐ m-ric/ai-travel-planner
Thank you @freddyaboulton for the ๐๐๐๐๐๐_๐๐๐๐๐๐ component, and @clem , @pngwn , @abidlabs for your ideas and support!
I wanted to try out if a powerful LLM like Mixtral-8x7b had geographical reasoning capabilities.
So I built a small space that prompts the LLM to provide a JSON list of places based on a user input.
And the result was impressive! ๐คฏ
โ ๐๐ ๐๐ฒ๐ฒ๐บ๐ ๐น๐ถ๐ธ๐ฒ ๐ ๐ถ๐ ๐๐ฟ๐ฎ๐น ๐ต๐ฎ๐ ๐ฎ ๐ด๐ฟ๐ฎ๐๐ฝ ๐ผ๐ณ ๐ด๐ฒ๐ผ๐ด๐ฟ๐ฎ๐ฝ๐ต๐ถ๐ฐ๐ฎ๐น ๐ฐ๐ผ๐ป๐ฐ๐ฒ๐ฝ๐๐ ๐น๐ถ๐ธ๐ฒ ๐ก๐ผ๐ฟ๐๐ต - ๐ฆ๐ผ๐๐๐ต, ๐ผ๐ฟ ๐๐ฝ๐ฎ๐๐ถ๐ฎ๐น ๐ฎ๐น๐ถ๐ด๐ป๐บ๐ฒ๐ป๐.๐งญ Not just describing these concepts, but really applying them in practice, for instance to successfully answer "give me 4 European cities that are aligned on the map". This is a ๐ป๐ถ๐ฐ๐ฒ ๐ฒ๐ ๐ฎ๐บ๐ฝ๐น๐ฒ ๐ผ๐ณ ๐ฎ๐ป ๐ฒ๐บ๐ฒ๐ฟ๐ด๐ฒ๐ป๐ ๐ฐ๐ฎ๐ฝ๐ฎ๐ฏ๐ถ๐น๐ถ๐๐, since nothing in the LLM's training data should prepare it for this specific task.
Anyway, I added API calls and a nice visualization on top of the LLM, streaming output, caching for the answers and locations... and ta-da! โจ I got the ๐๐ ๐ง๐ฟ๐ฎ๐๐ฒ๐น ๐ฃ๐น๐ฎ๐ป๐ป๐ฒ๐ฟ.
๐๐ค๐ช ๐๐๐ฃ ๐๐๐จ๐๐ง๐๐๐ ๐๐ฉ ๐ฎ๐ค๐ช๐ง ๐ฉ๐ง๐๐ฅ, ๐๐ฃ๐ ๐๐ฉ ๐ฌ๐๐ก๐ก ๐๐ค๐ข๐ ๐ช๐ฅ ๐ฌ๐๐ฉ๐ ๐ฃ๐๐๐ ๐๐ฃ๐ ๐๐ค๐ฃ๐ซ๐๐ฃ๐๐๐ฃ๐ฉ ๐ก๐ค๐๐๐ฉ๐๐ค๐ฃ๐จ!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐ ๐ m-ric/ai-travel-planner
Thank you @freddyaboulton for the ๐๐๐๐๐๐_๐๐๐๐๐๐ component, and @clem , @pngwn , @abidlabs for your ideas and support!
- 1 reply
posted
an
update
about 1 month ago
Post
2035
[๐๐๐ฐ ๐๐๐ฉ๐๐ซ] ๐๐ฅ๐ฅ ๐ญ๐จ๐ค๐๐ง๐ฌ ๐ฌ๐ก๐จ๐ฎ๐ฅ๐ ๐ง๐จ๐ญ ๐ซ๐๐ช๐ฎ๐ข๐ซ๐ ๐ญ๐ก๐ ๐ฌ๐๐ฆ๐ ๐๐๐๐จ๐ซ๐ญ ๐ญ๐จ ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐! โ ๐๐ข๐ฑ๐ญ๐ฎ๐ซ๐ ๐จ๐ ๐๐๐ฉ๐ญ๐ก๐ฌ ๐ซง๐
Google Researchers were unhappy with the way current decoding generally works: all tokens go through the same layers, thus requiring exactly the same effort to compute.
Whereas in reality, completing the answer to a difficult math problem for instance should be more computationally intense than completing the text of the Declaration of Independence: ๐ป๐ผ๐ ๐ฎ๐น๐น ๐๐ผ๐ธ๐ฒ๐ป๐ ๐ฎ๐ฟ๐ฒ ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ๐ฑ ๐ฒ๐พ๐๐ฎ๐น!
โก๏ธ ๐ง๐ต๐ฒ๐ ๐ต๐ฎ๐ฑ ๐๐ต๐ถ๐ ๐ด๐ฒ๐ป๐ถ๐๐ ๐ถ๐ฑ๐ฒ๐ฎ: ๐ก ๐ต๐ฎ๐๐ถ๐ป๐ด ๐ฎ ๐๐ผ๐ธ๐ฒ๐ป ๐ด๐ผ ๐๐ต๐ฟ๐ผ๐๐ด๐ต ๐ฎ ๐ฏ๐น๐ผ๐ฐ๐ธ ๐๐ต๐ผ๐๐น๐ฑ ๐ฏ๐ฒ ๐ผ๐ฝ๐๐ถ๐ผ๐ป๐ฎ๐น. The token can go through the block (thus undergoing expensive self-attention computation) or avoid it through a skip connection.
The routing decision is taken on the block level: each block selects from the total sequence the top-k tokens that will go through it, and the others tokens will skip it. ๐๐ฉ๐ช๐ด ๐ข๐ญ๐ญ๐ฐ๐ธ๐ด ๐ต๐ฐ ๐ค๐ฉ๐ฐ๐ฐ๐ด๐ฆ ๐ต๐ฉ๐ฆ ๐ฆ๐น๐ข๐ค๐ต ๐๐๐ฅ๐๐๐๐ฉ๐ฎ ๐ฐ๐ง ๐ข ๐ฃ๐ญ๐ฐ๐ค๐ฌ, ๐ช.๐ฆ. ๐ต๐ฉ๐ฆ ๐ฑ๐ณ๐ฐ๐ฑ๐ฐ๐ณ๐ต๐ช๐ฐ๐ฏ ๐ฐ๐ง ๐ต๐ฐ๐ฌ๐ฆ๐ฏ๐ด ๐ต๐ฉ๐ข๐ต ๐จ๐ฐ ๐ต๐ฉ๐ณ๐ฐ๐ถ๐จ๐ฉ ๐ช๐ต, ๐ธ๐ฉ๐ช๐ค๐ฉ ๐ฅ๐ช๐ณ๐ฆ๐ค๐ต๐ญ๐บ ๐ช๐ฏ๐ง๐ญ๐ถ๐ฆ๐ฏ๐ค๐ฆ๐ด ๐ต๐ฉ๐ฆ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ข๐ต๐ช๐ฐ๐ฏ๐ข๐ญ ๐ช๐ฏ๐ต๐ฆ๐ฏ๐ด๐ช๐ต๐บ ๐ฐ๐ง ๐ต๐ฉ๐ฆ ๐ง๐ฐ๐ณ๐ธ๐ข๐ณ๐ฅ ๐ฑ๐ข๐ด๐ด.
This yields Mixture-of-Depths (MoD), with spectacular results.
โจ ๐ฅ๐ฒ๐๐๐น๐๐:
๐๏ธ ๐๐ฎ๐ฝ๐ฎ๐ฐ๐ถ๐๐ ๐ฐ๐ฎ๐ป ๐ฏ๐ฒ ๐๐๐ป๐ฒ๐ฑ ๐ฎ๐น๐น ๐๐ต๐ฒ ๐๐ฎ๐ ๐ฑ๐ผ๐๐ป ๐๐ผ ๐ญ๐ฎ.๐ฑ% for every second block: thus 87.5% of tokens just skip the block!
๐ For the same training time and performance, >๐ฒ๐ฌ% ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ ๐๐ฝ๐ฒ๐ฒ๐ฑ!
๐ค ๐๐ฎ๐ป ๐ฏ๐ฒ ๐ฐ๐ผ๐บ๐ฏ๐ถ๐ป๐ฒ๐ฑ ๐๐ถ๐๐ต ๐ ๐ถ๐ ๐๐๐ฟ๐ฒ-๐ผ๐ณ-๐๐ ๐ฝ๐ฒ๐ฟ๐๐ for further improvements.
๐ ๐ฃ๐ฎ๐ฝ๐ฒ๐ฟ ๐ต๐ฒ๐ฟ๐ฒ ๐ Mixture-of-Depths: Dynamically allocating compute in transformer-based language models (2404.02258)
๐ I added it to my paper collection ๐ m-ric/spinning-up-in-llms-659e698f9dd5a71bd3f579a7
Google Researchers were unhappy with the way current decoding generally works: all tokens go through the same layers, thus requiring exactly the same effort to compute.
Whereas in reality, completing the answer to a difficult math problem for instance should be more computationally intense than completing the text of the Declaration of Independence: ๐ป๐ผ๐ ๐ฎ๐น๐น ๐๐ผ๐ธ๐ฒ๐ป๐ ๐ฎ๐ฟ๐ฒ ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ๐ฑ ๐ฒ๐พ๐๐ฎ๐น!
โก๏ธ ๐ง๐ต๐ฒ๐ ๐ต๐ฎ๐ฑ ๐๐ต๐ถ๐ ๐ด๐ฒ๐ป๐ถ๐๐ ๐ถ๐ฑ๐ฒ๐ฎ: ๐ก ๐ต๐ฎ๐๐ถ๐ป๐ด ๐ฎ ๐๐ผ๐ธ๐ฒ๐ป ๐ด๐ผ ๐๐ต๐ฟ๐ผ๐๐ด๐ต ๐ฎ ๐ฏ๐น๐ผ๐ฐ๐ธ ๐๐ต๐ผ๐๐น๐ฑ ๐ฏ๐ฒ ๐ผ๐ฝ๐๐ถ๐ผ๐ป๐ฎ๐น. The token can go through the block (thus undergoing expensive self-attention computation) or avoid it through a skip connection.
The routing decision is taken on the block level: each block selects from the total sequence the top-k tokens that will go through it, and the others tokens will skip it. ๐๐ฉ๐ช๐ด ๐ข๐ญ๐ญ๐ฐ๐ธ๐ด ๐ต๐ฐ ๐ค๐ฉ๐ฐ๐ฐ๐ด๐ฆ ๐ต๐ฉ๐ฆ ๐ฆ๐น๐ข๐ค๐ต ๐๐๐ฅ๐๐๐๐ฉ๐ฎ ๐ฐ๐ง ๐ข ๐ฃ๐ญ๐ฐ๐ค๐ฌ, ๐ช.๐ฆ. ๐ต๐ฉ๐ฆ ๐ฑ๐ณ๐ฐ๐ฑ๐ฐ๐ณ๐ต๐ช๐ฐ๐ฏ ๐ฐ๐ง ๐ต๐ฐ๐ฌ๐ฆ๐ฏ๐ด ๐ต๐ฉ๐ข๐ต ๐จ๐ฐ ๐ต๐ฉ๐ณ๐ฐ๐ถ๐จ๐ฉ ๐ช๐ต, ๐ธ๐ฉ๐ช๐ค๐ฉ ๐ฅ๐ช๐ณ๐ฆ๐ค๐ต๐ญ๐บ ๐ช๐ฏ๐ง๐ญ๐ถ๐ฆ๐ฏ๐ค๐ฆ๐ด ๐ต๐ฉ๐ฆ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ข๐ต๐ช๐ฐ๐ฏ๐ข๐ญ ๐ช๐ฏ๐ต๐ฆ๐ฏ๐ด๐ช๐ต๐บ ๐ฐ๐ง ๐ต๐ฉ๐ฆ ๐ง๐ฐ๐ณ๐ธ๐ข๐ณ๐ฅ ๐ฑ๐ข๐ด๐ด.
This yields Mixture-of-Depths (MoD), with spectacular results.
โจ ๐ฅ๐ฒ๐๐๐น๐๐:
๐๏ธ ๐๐ฎ๐ฝ๐ฎ๐ฐ๐ถ๐๐ ๐ฐ๐ฎ๐ป ๐ฏ๐ฒ ๐๐๐ป๐ฒ๐ฑ ๐ฎ๐น๐น ๐๐ต๐ฒ ๐๐ฎ๐ ๐ฑ๐ผ๐๐ป ๐๐ผ ๐ญ๐ฎ.๐ฑ% for every second block: thus 87.5% of tokens just skip the block!
๐ For the same training time and performance, >๐ฒ๐ฌ% ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ ๐๐ฝ๐ฒ๐ฒ๐ฑ!
๐ค ๐๐ฎ๐ป ๐ฏ๐ฒ ๐ฐ๐ผ๐บ๐ฏ๐ถ๐ป๐ฒ๐ฑ ๐๐ถ๐๐ต ๐ ๐ถ๐ ๐๐๐ฟ๐ฒ-๐ผ๐ณ-๐๐ ๐ฝ๐ฒ๐ฟ๐๐ for further improvements.
๐ ๐ฃ๐ฎ๐ฝ๐ฒ๐ฟ ๐ต๐ฒ๐ฟ๐ฒ ๐ Mixture-of-Depths: Dynamically allocating compute in transformer-based language models (2404.02258)
๐ I added it to my paper collection ๐ m-ric/spinning-up-in-llms-659e698f9dd5a71bd3f579a7
- 1 reply
posted
an
update
about 1 month ago
Post
1805
๐๐๐๐, ๐ญ๐ก๐ ๐ฒ๐๐๐ซ ๐จ๐ ๐๐ ๐๐ง๐ญ ๐ฐ๐จ๐ซ๐ค๐๐ฅ๐จ๐ฐ๐ฌ ๐ง๐ฆพ๐ค
I've just watched Andrew Ng's talk at Sequoia last week.
If you're interested in Agents, you should really watch it!
๐ช๐ต๐ ๐๐๐ฒ ๐ฎ๐ด๐ฒ๐ป๐ ๐๐ผ๐ฟ๐ธ๐ณ๐น๐ผ๐๐?
The current LLM task solving workflow is not very intuitive:
We ask it โwrite an essay all in one shot, without ever using backspace.โ
Why not allow the LLM a more similar process to what we would do?
- โWrite an essay outlineโ
- โDo you need wen research?โ
- โWrite a first draftโ
- โConsider improvementsโ
โฆ
This is called an Agentic workflow. Existing ones bring a huge performance boost. With HumanEval: GPT-4 zero-shot gets 67% score, agentic with either one of tool use or reflection goes over 90%, and the combination of the two scores even higher!
๐๐ด๐ฒ๐ป๐๐ถ๐ฐ ๐ฟ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด ๐ฑ๐ฒ๐๐ถ๐ด๐ป ๐ฝ๐ฎ๐๐๐ฒ๐ฟ๐ป๐
On the following two points, the tech is robust:
โ๏ธ ๐ฅ๐ฒ๐ณ๐น๐ฒ๐ ๐ถ๐ผ๐ป: For instance: add a critic step after the writing step
๐ ๏ธ ๐ง๐ผ๐ผ๐น ๐๐๐ฒ: extends the capabilities of the LLM by allowing it to call tools, like search or calculator
The next two will be needed to go further, but the tech for them is more emerging and not reliable yet:
๐บ๏ธ ๐ฃ๐น๐ฎ๐ป๐ป๐ถ๐ป๐ด forward to decompose task into subtasks. This allows great behaviours like an AI Agent re-routing after a failure
๐ ๐ ๐๐น๐๐ถ-๐ฎ๐ด๐ฒ๐ป๐ ๐ฐ๐ผ๐น๐น๐ฎ๐ฏ๐ผ๐ฟ๐ฎ๐๐ถ๐ผ๐ป: Program a flock of agents with tasks.
Improving the two above points will unlock huge performance boosts!
Andrew NG says Research agents are already part of his workflow!
๐๐น๐ผ๐๐ถ๐ป๐ด ๐๐ต๐ผ๐๐ด๐ต๐๐
Andrew speculates that through agentic workflows, maybe generating many tokens fast from a small LLM will give better results than slower throughput from a powerful LLM like GPT-5.
๐ฌ Watch the talk here ๐ https://www.youtube.com/watch?v=sal78ACtGTc
๐ I've added his recommended reads to m-ric/agents-65ba776fbd9e29f771c07d4e
I've just watched Andrew Ng's talk at Sequoia last week.
If you're interested in Agents, you should really watch it!
๐ช๐ต๐ ๐๐๐ฒ ๐ฎ๐ด๐ฒ๐ป๐ ๐๐ผ๐ฟ๐ธ๐ณ๐น๐ผ๐๐?
The current LLM task solving workflow is not very intuitive:
We ask it โwrite an essay all in one shot, without ever using backspace.โ
Why not allow the LLM a more similar process to what we would do?
- โWrite an essay outlineโ
- โDo you need wen research?โ
- โWrite a first draftโ
- โConsider improvementsโ
โฆ
This is called an Agentic workflow. Existing ones bring a huge performance boost. With HumanEval: GPT-4 zero-shot gets 67% score, agentic with either one of tool use or reflection goes over 90%, and the combination of the two scores even higher!
๐๐ด๐ฒ๐ป๐๐ถ๐ฐ ๐ฟ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด ๐ฑ๐ฒ๐๐ถ๐ด๐ป ๐ฝ๐ฎ๐๐๐ฒ๐ฟ๐ป๐
On the following two points, the tech is robust:
โ๏ธ ๐ฅ๐ฒ๐ณ๐น๐ฒ๐ ๐ถ๐ผ๐ป: For instance: add a critic step after the writing step
๐ ๏ธ ๐ง๐ผ๐ผ๐น ๐๐๐ฒ: extends the capabilities of the LLM by allowing it to call tools, like search or calculator
The next two will be needed to go further, but the tech for them is more emerging and not reliable yet:
๐บ๏ธ ๐ฃ๐น๐ฎ๐ป๐ป๐ถ๐ป๐ด forward to decompose task into subtasks. This allows great behaviours like an AI Agent re-routing after a failure
๐ ๐ ๐๐น๐๐ถ-๐ฎ๐ด๐ฒ๐ป๐ ๐ฐ๐ผ๐น๐น๐ฎ๐ฏ๐ผ๐ฟ๐ฎ๐๐ถ๐ผ๐ป: Program a flock of agents with tasks.
Improving the two above points will unlock huge performance boosts!
Andrew NG says Research agents are already part of his workflow!
๐๐น๐ผ๐๐ถ๐ป๐ด ๐๐ต๐ผ๐๐ด๐ต๐๐
Andrew speculates that through agentic workflows, maybe generating many tokens fast from a small LLM will give better results than slower throughput from a powerful LLM like GPT-5.
๐ฌ Watch the talk here ๐ https://www.youtube.com/watch?v=sal78ACtGTc
๐ I've added his recommended reads to m-ric/agents-65ba776fbd9e29f771c07d4e
- 1 reply
posted
an
update
about 1 month ago
Post
1768
๐๐ก๐ ๐ซ๐๐ญ๐ฎ๐ซ๐ง ๐จ๐ ๐ญ๐ก๐ ๐๐๐๐ฌ โ ๐๐๐ฐ ๐๐๐ฆ๐๐-๐๐๐ฌ๐๐ ๐๐ซ๐๐ก๐ข๐ญ๐๐๐ญ๐ฎ๐ซ๐ "๐๐๐ฆ๐๐"
Since the release of BERT by Google in 2019, Transformers architecture have taken over machine learning thanks to their ๐ฎ๐๐๐ฒ๐ป๐๐ถ๐ผ๐ป ๐บ๐ฒ๐ฐ๐ต๐ฎ๐ป๐ถ๐๐บ, that gives them the ability to focus on important points of the input. But ๐๐ฉ๐ฉ๐๐ฃ๐ฉ๐๐ค๐ฃ ๐๐ค๐ข๐ฅ๐ช๐ฉ๐๐ฉ๐๐ค๐ฃ ๐๐จ ๐ฆ๐ช๐๐๐ง๐๐ฉ๐๐ ๐๐ฃ ๐ฉ๐๐ ๐๐ฃ๐ฅ๐ช๐ฉ ๐ก๐๐ฃ๐๐ฉ๐.
๐ซ The Mamba paper, published in December 2023, announced the return of the RNNs: it has no attention, but integrates a selection mechanism, which should be able to reproduce the โfocusโ ability of attention, in an architecture for which the compute requirements ๐ด๐ฟ๐ผ๐ ๐ผ๐ป๐น๐ ๐น๐ถ๐ป๐ฒ๐ฎ๐ฟ๐น๐ ๐ถ๐ป ๐ถ๐ป๐ฝ๐๐ ๐น๐ฒ๐ป๐ด๐๐ต!
๐ค Would this work? We had yet to see a large Mamba model recovering the performance of Attention-based Transformers.
๐ฅ But now it's done! A (Mamba + Transformers) hybrid just beat Transformers!
The AI21 Labs team just released Jamba.
They insert a few Transformer layers to inject some attention in a big pile of Mamba layers, thus getting the best of both worlds.
๐๐;๐ฟ๐:
๐๏ธ ๐ก๐ฒ๐ ๐ ๐ผ๐ ๐ฎ๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ: 4 Jamba blocks, each of these being 7 Mamba layers for 1 Transformer.
๐๏ธ ๐ฑ๐ฎ๐ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐, ๐ญ๐ฎ๐ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฎ๐ ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ: This reduction is enabled by Mixture of Experts, and similar to Mixtral (47B parameters - 13B active).
๐๏ธ ๐ฆ๐ฝ๐ฒ๐ฒ๐ฑ: ๐ ๐ฏ ๐๐ต๐ฟ๐ผ๐๐ด๐ต๐ฝ๐๐. Jamba is much faster than similar-sized Transformer models on long contexts.
๐ ๐๐ผ๐ป๐๐ฒ๐ ๐ ๐น๐ฒ๐ป๐ด๐๐ต: ๐ญ๐ฐ๐ฌ๐ ๐๐ผ๐ธ๐ฒ๐ป๐ on a single 80GB A100!
๐ช ๐ฃ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ: ๐๐๐ฎ๐๐ฒ-๐ผ๐ณ-๐๐ต๐ฒ-๐ฎ๐ฟ๐ ๐ณ๐ผ๐ฟ ๐๐ต๐ถ๐ ๐๐ถ๐๐ฒ. The small injection of attention seems sufficient since Jamba beats the open-source reference Mixtral-8x7B on many benchmarks!
Try it here ๐ ai21labs/Jamba-v0.1
Since the release of BERT by Google in 2019, Transformers architecture have taken over machine learning thanks to their ๐ฎ๐๐๐ฒ๐ป๐๐ถ๐ผ๐ป ๐บ๐ฒ๐ฐ๐ต๐ฎ๐ป๐ถ๐๐บ, that gives them the ability to focus on important points of the input. But ๐๐ฉ๐ฉ๐๐ฃ๐ฉ๐๐ค๐ฃ ๐๐ค๐ข๐ฅ๐ช๐ฉ๐๐ฉ๐๐ค๐ฃ ๐๐จ ๐ฆ๐ช๐๐๐ง๐๐ฉ๐๐ ๐๐ฃ ๐ฉ๐๐ ๐๐ฃ๐ฅ๐ช๐ฉ ๐ก๐๐ฃ๐๐ฉ๐.
๐ซ The Mamba paper, published in December 2023, announced the return of the RNNs: it has no attention, but integrates a selection mechanism, which should be able to reproduce the โfocusโ ability of attention, in an architecture for which the compute requirements ๐ด๐ฟ๐ผ๐ ๐ผ๐ป๐น๐ ๐น๐ถ๐ป๐ฒ๐ฎ๐ฟ๐น๐ ๐ถ๐ป ๐ถ๐ป๐ฝ๐๐ ๐น๐ฒ๐ป๐ด๐๐ต!
๐ค Would this work? We had yet to see a large Mamba model recovering the performance of Attention-based Transformers.
๐ฅ But now it's done! A (Mamba + Transformers) hybrid just beat Transformers!
The AI21 Labs team just released Jamba.
They insert a few Transformer layers to inject some attention in a big pile of Mamba layers, thus getting the best of both worlds.
๐๐;๐ฟ๐:
๐๏ธ ๐ก๐ฒ๐ ๐ ๐ผ๐ ๐ฎ๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ: 4 Jamba blocks, each of these being 7 Mamba layers for 1 Transformer.
๐๏ธ ๐ฑ๐ฎ๐ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐, ๐ญ๐ฎ๐ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฎ๐ ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ: This reduction is enabled by Mixture of Experts, and similar to Mixtral (47B parameters - 13B active).
๐๏ธ ๐ฆ๐ฝ๐ฒ๐ฒ๐ฑ: ๐ ๐ฏ ๐๐ต๐ฟ๐ผ๐๐ด๐ต๐ฝ๐๐. Jamba is much faster than similar-sized Transformer models on long contexts.
๐ ๐๐ผ๐ป๐๐ฒ๐ ๐ ๐น๐ฒ๐ป๐ด๐๐ต: ๐ญ๐ฐ๐ฌ๐ ๐๐ผ๐ธ๐ฒ๐ป๐ on a single 80GB A100!
๐ช ๐ฃ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ: ๐๐๐ฎ๐๐ฒ-๐ผ๐ณ-๐๐ต๐ฒ-๐ฎ๐ฟ๐ ๐ณ๐ผ๐ฟ ๐๐ต๐ถ๐ ๐๐ถ๐๐ฒ. The small injection of attention seems sufficient since Jamba beats the open-source reference Mixtral-8x7B on many benchmarks!
Try it here ๐ ai21labs/Jamba-v0.1
posted
an
update
about 1 month ago
Post
1664
๐๐ผ๐ ๐ฑ๐ผ๐ฒ๐ ๐ฏ๐ฒ๐ฎ๐บ ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐๐ผ๐ฟ๐ธ? โก๏ธ ๐๐๐ฌ ๐ซ๐๐จ๐ช๐๐ก๐๐ฏ๐๐ฉ๐๐ค๐ฃ ๐ฉ๐ค๐ค๐ก! ๐
In Decoder-type LLMs like GPT4 or Mistral-Large, the output is generated one token (=word part) at a time. That's why they're nicknamed "stochastic parrots": the "thinking" process only happens one step at a time, so it can seem really myopic.
๐๐จ ๐ก๐จ๐ฐ ๐ข๐ฌ ๐ญ๐ก๐ ๐ง๐๐ฑ๐ญ ๐ญ๐จ๐ค๐๐ง ๐ฌ๐๐ฅ๐๐๐ญ๐๐?
๐ Given its input sentence like "๐๐ฉ๐ข๐ต ๐ช๐ด ๐ต๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ? ๐๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ", the Decoder LLM generates, for each token in its vocabulary, a score that represents this token's probability of coming next.
For instance: "๐๐จ" gets score 0.56, and "๐๐๐ฃ" gets score 0.35.
๐ค ๐๐ซ๐๐๐๐ฒ ๐๐๐๐จ๐๐ข๐ง๐ is the naive option where you simply take the next most probable token at each step. But this creates paths that maximize very short-term rewards, thus may overlook better paths for the long term (like this time when you played FIFA all evening and arrived unprepared to your school exam on the next day).
In our example, the next highest score token might be "๐๐จ", but this will strongly bias the LLM towards giving an hasty response. On the opposite, starting with "๐๐๐ฃ" could have been completed with "๐ฃ๐ฆ ๐ฐ๐ฃ๐ต๐ข๐ช๐ฏ๐ฆ๐ฅ ๐ง๐ณ๐ฐ๐ฎ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ช๐ฏ๐จ ๐ฑ๐ณ๐ฆ๐ท๐ช๐ฐ๐ถ๐ด ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ๐ด ๐ง๐ช๐ณ๐ด๐ต", which steers the LLM towards a correct reasoning!
๐บ๏ธ ๐๐๐๐ฆ ๐ฌ๐๐๐ซ๐๐ก improves on greedy decoding by generating at each step several paths - called beams - instead of one. This allows the generation to explore a much larger space, thus find better completions. In our example, both the "๐๐จ" and the "๐๐๐ฃ" completion could be tested. โ
๐ I've created a tool to let you visualize it, thank you @joaogante for your great help!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐: m-ric/beam_search_visualizer
In Decoder-type LLMs like GPT4 or Mistral-Large, the output is generated one token (=word part) at a time. That's why they're nicknamed "stochastic parrots": the "thinking" process only happens one step at a time, so it can seem really myopic.
๐๐จ ๐ก๐จ๐ฐ ๐ข๐ฌ ๐ญ๐ก๐ ๐ง๐๐ฑ๐ญ ๐ญ๐จ๐ค๐๐ง ๐ฌ๐๐ฅ๐๐๐ญ๐๐?
๐ Given its input sentence like "๐๐ฉ๐ข๐ต ๐ช๐ด ๐ต๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ? ๐๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ", the Decoder LLM generates, for each token in its vocabulary, a score that represents this token's probability of coming next.
For instance: "๐๐จ" gets score 0.56, and "๐๐๐ฃ" gets score 0.35.
๐ค ๐๐ซ๐๐๐๐ฒ ๐๐๐๐จ๐๐ข๐ง๐ is the naive option where you simply take the next most probable token at each step. But this creates paths that maximize very short-term rewards, thus may overlook better paths for the long term (like this time when you played FIFA all evening and arrived unprepared to your school exam on the next day).
In our example, the next highest score token might be "๐๐จ", but this will strongly bias the LLM towards giving an hasty response. On the opposite, starting with "๐๐๐ฃ" could have been completed with "๐ฃ๐ฆ ๐ฐ๐ฃ๐ต๐ข๐ช๐ฏ๐ฆ๐ฅ ๐ง๐ณ๐ฐ๐ฎ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ช๐ฏ๐จ ๐ฑ๐ณ๐ฆ๐ท๐ช๐ฐ๐ถ๐ด ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ๐ด ๐ง๐ช๐ณ๐ด๐ต", which steers the LLM towards a correct reasoning!
๐บ๏ธ ๐๐๐๐ฆ ๐ฌ๐๐๐ซ๐๐ก improves on greedy decoding by generating at each step several paths - called beams - instead of one. This allows the generation to explore a much larger space, thus find better completions. In our example, both the "๐๐จ" and the "๐๐๐ฃ" completion could be tested. โ
๐ I've created a tool to let you visualize it, thank you @joaogante for your great help!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐: m-ric/beam_search_visualizer
posted
an
update
about 2 months ago
Post
2011
๐จ๐๐ถ๐ป๐ด ๐๐๐ -๐ฎ๐-๐ฎ-๐ท๐๐ฑ๐ด๐ฒ ๐งโโ๏ธ ๐ณ๐ผ๐ฟ ๐ฎ๐ป ๐ฎ๐๐๐ผ๐บ๐ฎ๐๐ฒ๐ฑ ๐ฎ๐ป๐ฑ ๐๐ฒ๐ฟ๐๐ฎ๐๐ถ๐น๐ฒ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ถ๐ผ๐ป
Evaluating LLM outputs is often hard, since many tasks require open-ended answers for which no deterministic metrics work: for instance, when asking a model to summarize a text, there could be hundreds of correct ways to do it. The most versatile way to grade these outputs is then human evaluation, but it is very time-consuming, thus costly.
๐ค Then ๐๐ต๐ ๐ป๐ผ๐ ๐ฎ๐๐ธ ๐ฎ๐ป๐ผ๐๐ต๐ฒ๐ฟ ๐๐๐ ๐๐ผ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ถ๐ผ๐ป, by providing it relevant rating criteria? ๐ This is the idea behind LLM-as-a-judge.
โ๏ธ To implement a LLM judge correctly, you need a few tricks.
โ So ๐'๐๐ฒ ๐ท๐๐๐ ๐ฝ๐๐ฏ๐น๐ถ๐๐ต๐ฒ๐ฑ ๐ฎ ๐ป๐ฒ๐ ๐ป๐ผ๐๐ฒ๐ฏ๐ผ๐ผ๐ธ ๐๐ต๐ผ๐๐ถ๐ป๐ด ๐ต๐ผ๐ ๐๐ผ ๐ถ๐บ๐ฝ๐น๐ฒ๐บ๐ฒ๐ป๐ ๐ถ๐ ๐ฝ๐ฟ๐ผ๐ฝ๐ฒ๐ฟ๐น๐ ๐ถ๐ป ๐ผ๐๐ฟ ๐๐๐ด๐ด๐ถ๐ป๐ด ๐๐ฎ๐ฐ๐ฒ ๐๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ! (you can run it instantly in Google Colab)
โก๏ธ ๐๐๐ -๐ฎ๐-๐ฎ-๐ท๐๐ฑ๐ด๐ฒ ๐ฐ๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ: https://huggingface.co/learn/cookbook/llm_judge
The Cookbook is a great collection of notebooks demonstrating recipes (thus the "cookbook") for common LLM usages. I recommend you to go take a look!
โก๏ธ ๐๐น๐น ๐ฐ๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ๐: https://huggingface.co/learn/cookbook/index
Thank you @MariaK for your support!
Evaluating LLM outputs is often hard, since many tasks require open-ended answers for which no deterministic metrics work: for instance, when asking a model to summarize a text, there could be hundreds of correct ways to do it. The most versatile way to grade these outputs is then human evaluation, but it is very time-consuming, thus costly.
๐ค Then ๐๐ต๐ ๐ป๐ผ๐ ๐ฎ๐๐ธ ๐ฎ๐ป๐ผ๐๐ต๐ฒ๐ฟ ๐๐๐ ๐๐ผ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ถ๐ผ๐ป, by providing it relevant rating criteria? ๐ This is the idea behind LLM-as-a-judge.
โ๏ธ To implement a LLM judge correctly, you need a few tricks.
โ So ๐'๐๐ฒ ๐ท๐๐๐ ๐ฝ๐๐ฏ๐น๐ถ๐๐ต๐ฒ๐ฑ ๐ฎ ๐ป๐ฒ๐ ๐ป๐ผ๐๐ฒ๐ฏ๐ผ๐ผ๐ธ ๐๐ต๐ผ๐๐ถ๐ป๐ด ๐ต๐ผ๐ ๐๐ผ ๐ถ๐บ๐ฝ๐น๐ฒ๐บ๐ฒ๐ป๐ ๐ถ๐ ๐ฝ๐ฟ๐ผ๐ฝ๐ฒ๐ฟ๐น๐ ๐ถ๐ป ๐ผ๐๐ฟ ๐๐๐ด๐ด๐ถ๐ป๐ด ๐๐ฎ๐ฐ๐ฒ ๐๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ! (you can run it instantly in Google Colab)
โก๏ธ ๐๐๐ -๐ฎ๐-๐ฎ-๐ท๐๐ฑ๐ด๐ฒ ๐ฐ๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ: https://huggingface.co/learn/cookbook/llm_judge
The Cookbook is a great collection of notebooks demonstrating recipes (thus the "cookbook") for common LLM usages. I recommend you to go take a look!
โก๏ธ ๐๐น๐น ๐ฐ๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ๐: https://huggingface.co/learn/cookbook/index
Thank you @MariaK for your support!
- 2 replies
posted
an
update
about 2 months ago
Post
Interesting paper: ๐๐๐๐จ๐ซ๐: ๐ญ๐ซ๐๐ข๐ง ๐๐ ๐ฆ๐จ๐๐๐ฅ๐ฌ ๐จ๐ง ๐๐จ๐ง๐ฌ๐ฎ๐ฆ๐๐ซ-๐ ๐ซ๐๐๐ ๐๐๐๐ฌ ๐ช
It's now possible to ๐๐ช๐ก๐ก๐ฎ ๐ฅ๐ง๐-๐ฉ๐ง๐๐๐ฃ a 7B model on a consumer-grade GPU of 24Gb RAM, without any performance loss!
The memory usage of training models has always been an acute issue. For instance full pre-training of a 7B model used to eat ~50Gb of RAM!
The common workarounds to reduce memory load are:
- separate models on multiple GPUs ("sharding")
- quantize models: encode weights on fewer bits
Another technique is to ๐ฅ๐ง๐ค๐๐๐๐ฉ ๐ฉ๐๐ ๐ฌ๐๐๐๐๐ฉ ๐ข๐๐ฉ๐ง๐๐ญ ๐ฉ๐ค ๐ก๐ค๐ฌ๐๐ง-๐ง๐๐ฃ๐ ๐จ๐ฅ๐๐๐๐จ, (since sometimes the weights do not really vary on all dimensions): this can save a lot of space!
This low-rank projection can be done on adapters to preserve the original weights (go check out LoRA), but it still generally hurts the performance too much for pre-training.
โก๏ธ Enter the authors of ๐๐ข๐๐ฐ๐ณ๐ฆ: ๐๐ฆ๐ฎ๐ฐ๐ณ๐บ-๐๐ง๐ง๐ช๐ค๐ช๐ฆ๐ฏ๐ต ๐๐๐ ๐๐ณ๐ข๐ช๐ฏ๐ช๐ฏ๐จ ๐ฃ๐บ ๐๐ณ๐ข๐ฅ๐ช๐ฆ๐ฏ๐ต ๐๐ฐ๐ธ-๐๐ข๐ฏ๐ฌ ๐๐ณ๐ฐ๐ซ๐ฆ๐ค๐ต๐ช๐ฐ๐ฏ. They gather (and prove) interesting insights:
โ The weight matrix does not reliably converge to lower ranks during training.
โ But the gradient matrix does!
Based on these insights, ๐๐ต๐ฒ๐ ๐ฏ๐๐ถ๐น๐ฑ ๐๐ฎ๐๐ผ๐ฟ๐ฒ, that projects the gradient to lower ranks.
๐บ๏ธ ๐๐ฟ๐ฒ๐ฎ๐ ๐ถ๐ฑ๐ฒ๐ฎ: to leave the optimization free to explore more space, they periodically re-build the low-rank projection throughout the training (a nice illustration is in the paper).
๐ค This method can even be combined with previous ones such as 8-bit Adam (quantizing the optimizer states to 8-bit).
โก๏ธ ๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
๐ Of course, huge reduction in memory footprint allowing the training on consumer-grade GPU (cf figure).
๐ช No reduction in performance: this scales well up to 7B parameters (and was independently confirmed since) โ this is essential, it confirms that the method is viable!
Read the full paper here: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2403.03507)
It's now possible to ๐๐ช๐ก๐ก๐ฎ ๐ฅ๐ง๐-๐ฉ๐ง๐๐๐ฃ a 7B model on a consumer-grade GPU of 24Gb RAM, without any performance loss!
The memory usage of training models has always been an acute issue. For instance full pre-training of a 7B model used to eat ~50Gb of RAM!
The common workarounds to reduce memory load are:
- separate models on multiple GPUs ("sharding")
- quantize models: encode weights on fewer bits
Another technique is to ๐ฅ๐ง๐ค๐๐๐๐ฉ ๐ฉ๐๐ ๐ฌ๐๐๐๐๐ฉ ๐ข๐๐ฉ๐ง๐๐ญ ๐ฉ๐ค ๐ก๐ค๐ฌ๐๐ง-๐ง๐๐ฃ๐ ๐จ๐ฅ๐๐๐๐จ, (since sometimes the weights do not really vary on all dimensions): this can save a lot of space!
This low-rank projection can be done on adapters to preserve the original weights (go check out LoRA), but it still generally hurts the performance too much for pre-training.
โก๏ธ Enter the authors of ๐๐ข๐๐ฐ๐ณ๐ฆ: ๐๐ฆ๐ฎ๐ฐ๐ณ๐บ-๐๐ง๐ง๐ช๐ค๐ช๐ฆ๐ฏ๐ต ๐๐๐ ๐๐ณ๐ข๐ช๐ฏ๐ช๐ฏ๐จ ๐ฃ๐บ ๐๐ณ๐ข๐ฅ๐ช๐ฆ๐ฏ๐ต ๐๐ฐ๐ธ-๐๐ข๐ฏ๐ฌ ๐๐ณ๐ฐ๐ซ๐ฆ๐ค๐ต๐ช๐ฐ๐ฏ. They gather (and prove) interesting insights:
โ The weight matrix does not reliably converge to lower ranks during training.
โ But the gradient matrix does!
Based on these insights, ๐๐ต๐ฒ๐ ๐ฏ๐๐ถ๐น๐ฑ ๐๐ฎ๐๐ผ๐ฟ๐ฒ, that projects the gradient to lower ranks.
๐บ๏ธ ๐๐ฟ๐ฒ๐ฎ๐ ๐ถ๐ฑ๐ฒ๐ฎ: to leave the optimization free to explore more space, they periodically re-build the low-rank projection throughout the training (a nice illustration is in the paper).
๐ค This method can even be combined with previous ones such as 8-bit Adam (quantizing the optimizer states to 8-bit).
โก๏ธ ๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
๐ Of course, huge reduction in memory footprint allowing the training on consumer-grade GPU (cf figure).
๐ช No reduction in performance: this scales well up to 7B parameters (and was independently confirmed since) โ this is essential, it confirms that the method is viable!
Read the full paper here: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2403.03507)
posted
an
update
3 months ago
Post
๐๐ If you're building RAG applications, you should check this out:
โ๏ธ I've built a new space to let you visualize the chunks you get with different text splitting methods!
โก๏ธ Visualize your chunks here:
m-ric/chunk_visualizer
โ๏ธ I've built a new space to let you visualize the chunks you get with different text splitting methods!
โก๏ธ Visualize your chunks here:
m-ric/chunk_visualizer
- 2 replies