9 1 24

Alessandro Ercolani

giux78

https://alessandroercolani.webflow.io/

giux78

AI & ML interests

NLP, Reinforcement Learning, Semantics, Computational Neuroscience

Articles

Analysis on evaluating 7 bilions italian LLMs

9 days ago

• 2

Organizations

Posts 7

Post

🎉 Super @DeepMount00 just released 𝗚𝗲𝗺𝗺𝗮_𝗤𝗔_𝗜𝗧𝗔_𝘃𝟯 𝗹𝗲𝗮𝗱𝗶𝗻𝗴 the 𝗥𝗔𝗚 𝘁𝗮𝘀𝗸 on the Italian 𝗟𝗟𝗠_𝗜𝗧𝗔_𝗟𝗘𝗔𝗗𝗘𝗥𝗕𝗢𝗔𝗥𝗗. The model is a fine tuned version of Gemma 2B.
Model details: DeepMount00/Gemma_QA_ITA_v3
Explore the full RAG section rankings here: FinancialSupport/open_ita_llm_leaderboard on section Classifica RAG

Post

On evaluating fine tuned 7B Italian open source LLMs I have collected many data points and I created a super simple explorative analyses. My hypothesis based on data are:

- mmlu is hard to improve when fine tuning a base model on a different language
- fine tuning also on single GPUs can improve by 5% to 10% the base model on common tasks but a lot more on specific cases with the right training time and data
- fine tuning can specialize well but at cost of loosing some foundational knowledge.

Here the data https://docs.google.com/spreadsheets/d/1MBcxy1loK8eIycZG4DN84Q2ejZ0jSjxUBgoShHDR6IY/edit?usp=sharing
Here the colab https://colab.research.google.com/drive/1ra4_skG5QYWSYOzvagOoIoj4bibQD8Gw?usp=sharing
Here an article with some considerations https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1

View all posts