Analysis on evaluating 7 bilions italian LLMs
โข
2
Alessandro Ercolani
giux78
AI & ML interests
NLP, Reinforcement Learning, Semantics, Computational Neuroscience
Articles
Organizations
giux78's activity
posted
an
update
23 days ago
Post
1726
๐ Super
@DeepMount00
just released ๐๐ฒ๐บ๐บ๐ฎ_๐ค๐_๐๐ง๐_๐๐ฏ ๐น๐ฒ๐ฎ๐ฑ๐ถ๐ป๐ด the ๐ฅ๐๐ ๐๐ฎ๐๐ธ on the Italian ๐๐๐ _๐๐ง๐_๐๐๐๐๐๐ฅ๐๐ข๐๐ฅ๐. The model is a fine tuned version of Gemma 2B.
Model details: DeepMount00/Gemma_QA_ITA_v3
Explore the full RAG section rankings here: FinancialSupport/open_ita_llm_leaderboard on section Classifica RAG
Model details: DeepMount00/Gemma_QA_ITA_v3
Explore the full RAG section rankings here: FinancialSupport/open_ita_llm_leaderboard on section Classifica RAG
posted
an
update
about 1 month ago
Post
1749
On evaluating fine tuned 7B Italian open source LLMs I have collected many data points and I created a super simple explorative analyses. My hypothesis based on data are:
- mmlu is hard to improve when fine tuning a base model on a different language
- fine tuning also on single GPUs can improve by 5% to 10% the base model on common tasks but a lot more on specific cases with the right training time and data
- fine tuning can specialize well but at cost of loosing some foundational knowledge.
Here the data https://docs.google.com/spreadsheets/d/1MBcxy1loK8eIycZG4DN84Q2ejZ0jSjxUBgoShHDR6IY/edit?usp=sharing
Here the colab https://colab.research.google.com/drive/1ra4_skG5QYWSYOzvagOoIoj4bibQD8Gw?usp=sharing
Here an article with some considerations https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1
- mmlu is hard to improve when fine tuning a base model on a different language
- fine tuning also on single GPUs can improve by 5% to 10% the base model on common tasks but a lot more on specific cases with the right training time and data
- fine tuning can specialize well but at cost of loosing some foundational knowledge.
Here the data https://docs.google.com/spreadsheets/d/1MBcxy1loK8eIycZG4DN84Q2ejZ0jSjxUBgoShHDR6IY/edit?usp=sharing
Here the colab https://colab.research.google.com/drive/1ra4_skG5QYWSYOzvagOoIoj4bibQD8Gw?usp=sharing
Here an article with some considerations https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1
replied to
osanseviero's
post
about 1 month ago
posted
an
update
about 1 month ago
Post
1278
Based on the work of
@mrinaldi
and
@ruggsea
we just released the biggest - ready for training - conversational dataset based on Usenet data in the Italian language ๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น. It contains about 9 millions of conversations made by real humans.
mii-community/UsenetArchiveIT-conversations
mii-community/UsenetArchiveIT-conversations
replied to
their
post
about 1 month ago
@clefourrier sometimes rendering the data took lot of time as in the pic attached. But data are in a static a file inside the app. Is there a way to improve? FYI we are on the free tier.
replied to
their
post
about 1 month ago
sure cc @FinancialSupport
posted
an
update
about 1 month ago
Post
2652
I and
@FinancialSupport
just release the new Italian Leaderboard:
FinancialSupport/open_ita_llm_leaderboard
It is based on lm-evaluation-harness and at the moment , mainly, on 7 billion models. In the next weeks we will add more models. If you have suggestion or need explanations join our community discord https://discord.gg/a26cRkBCNH
FinancialSupport/open_ita_llm_leaderboard
It is based on lm-evaluation-harness and at the moment , mainly, on 7 billion models. In the next weeks we will add more models. If you have suggestion or need explanations join our community discord https://discord.gg/a26cRkBCNH
-
- 9 replies
posted
an
update
about 2 months ago
Post
Wonderful open source Italian dataset from
@manalog
and
@ruggsea
:
https://huggingface.co/datasets/manalog/UsenetArchiveIT
The dataset contributes to the https://huggingface.co/mii-community project, aimed at advancing the creation of Italian open-source Language Models (LLMs).๐ฎ๐น ๐ค About 10-20 billion token, probably the best conversational open source dataset in the Italian language. ๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น
https://huggingface.co/datasets/manalog/UsenetArchiveIT
The dataset contributes to the https://huggingface.co/mii-community project, aimed at advancing the creation of Italian open-source Language Models (LLMs).๐ฎ๐น ๐ค About 10-20 billion token, probably the best conversational open source dataset in the Italian language. ๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น
- 2 replies
replied to
their
post
about 2 months ago
Thanks a lot!!!!!!
posted
an
update
about 2 months ago
Post
Super work from
@DeepMount00
:
๐ ๐๐ข๐ฌ๐๐จ๐ฏ๐๐ซ ๐๐ง๐ข๐ฏ๐๐ซ๐ฌ๐๐ฅ ๐๐๐ซ: ๐ ๐๐ฅ๐ข๐๐๐ซ-๐๐๐ฌ๐๐ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐๐
Introducing ๐๐ง๐ข๐ฏ๐๐ซ๐ฌ๐๐ฅ ๐๐๐ซ ๐๐จ๐ซ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐๐ง๐ ๐ฎ๐๐ ๐, a revolutionary Named Entity Recognition (NER) model evolved from the GliNer architecture and meticulously tailored for the Italian language. This advanced model is a beacon of efficiency and versatility, engineered to ๐ซ๐๐๐จ๐ ๐ง๐ข๐ณ๐ ๐๐ง๐ฒ ๐๐ง๐ญ๐ข๐ญ๐ฒ ๐ญ๐ฒ๐ฉ๐ within the rich nuances of Italian, using a bidirectional transformer encoder. It stands out as an ideal solution for those navigating the challenges of resource-limited environments or seeking an efficient alternative to the cumbersome Large Language Models (LLMs).
๐๐ฎ๐ง๐ฌ ๐๐๐ฌ๐ญ ๐๐ฅ๐ฌ๐จ ๐จ๐ง ๐๐๐!
Experience this Italian-focused innovation live on Hugging Face Spaces:
DeepMount00/universal_ner_ita
Paper: https://arxiv.org/abs/2311.08526 Urchade Zaratiana et all. great work!
๐ ๐๐ข๐ฌ๐๐จ๐ฏ๐๐ซ ๐๐ง๐ข๐ฏ๐๐ซ๐ฌ๐๐ฅ ๐๐๐ซ: ๐ ๐๐ฅ๐ข๐๐๐ซ-๐๐๐ฌ๐๐ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐๐
Introducing ๐๐ง๐ข๐ฏ๐๐ซ๐ฌ๐๐ฅ ๐๐๐ซ ๐๐จ๐ซ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐๐ง๐ ๐ฎ๐๐ ๐, a revolutionary Named Entity Recognition (NER) model evolved from the GliNer architecture and meticulously tailored for the Italian language. This advanced model is a beacon of efficiency and versatility, engineered to ๐ซ๐๐๐จ๐ ๐ง๐ข๐ณ๐ ๐๐ง๐ฒ ๐๐ง๐ญ๐ข๐ญ๐ฒ ๐ญ๐ฒ๐ฉ๐ within the rich nuances of Italian, using a bidirectional transformer encoder. It stands out as an ideal solution for those navigating the challenges of resource-limited environments or seeking an efficient alternative to the cumbersome Large Language Models (LLMs).
๐๐ฎ๐ง๐ฌ ๐๐๐ฌ๐ญ ๐๐ฅ๐ฌ๐จ ๐จ๐ง ๐๐๐!
Experience this Italian-focused innovation live on Hugging Face Spaces:
DeepMount00/universal_ner_ita
Paper: https://arxiv.org/abs/2311.08526 Urchade Zaratiana et all. great work!
-
- 3 replies
posted
an
update
about 2 months ago
Post
The dataset of all Italian laws just released
mii-llm/gazzetta-ufficiale
Kudos to @efederici @mferraretto @z-uo
Kudos to @efederici @mferraretto @z-uo