|
--- |
|
library_name: peft |
|
model-index: |
|
- name: gembode-7b |
|
results: |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: ENEM Challenge (No Images) |
|
type: eduagarcia/enem_challenge |
|
split: train |
|
args: |
|
num_few_shot: 3 |
|
metrics: |
|
- type: acc |
|
value: 66.9 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: BLUEX (No Images) |
|
type: eduagarcia-temp/BLUEX_without_images |
|
split: train |
|
args: |
|
num_few_shot: 3 |
|
metrics: |
|
- type: acc |
|
value: 57.16 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: OAB Exams |
|
type: eduagarcia/oab_exams |
|
split: train |
|
args: |
|
num_few_shot: 3 |
|
metrics: |
|
- type: acc |
|
value: 45.47 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: Assin2 RTE |
|
type: assin2 |
|
split: test |
|
args: |
|
num_few_shot: 15 |
|
metrics: |
|
- type: f1_macro |
|
value: 86.61 |
|
name: f1-macro |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: Assin2 STS |
|
type: eduagarcia/portuguese_benchmark |
|
split: test |
|
args: |
|
num_few_shot: 15 |
|
metrics: |
|
- type: pearson |
|
value: 71.39 |
|
name: pearson |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: FaQuAD NLI |
|
type: ruanchaves/faquad-nli |
|
split: test |
|
args: |
|
num_few_shot: 15 |
|
metrics: |
|
- type: f1_macro |
|
value: 67.4 |
|
name: f1-macro |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: HateBR Binary |
|
type: ruanchaves/hatebr |
|
split: test |
|
args: |
|
num_few_shot: 25 |
|
metrics: |
|
- type: f1_macro |
|
value: 79.81 |
|
name: f1-macro |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: PT Hate Speech Binary |
|
type: hate_speech_portuguese |
|
split: test |
|
args: |
|
num_few_shot: 25 |
|
metrics: |
|
- type: f1_macro |
|
value: 63.75 |
|
name: f1-macro |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: tweetSentBR |
|
type: eduagarcia/tweetsentbr_fewshot |
|
split: test |
|
args: |
|
num_few_shot: 25 |
|
metrics: |
|
- type: f1_macro |
|
value: 65.49 |
|
name: f1-macro |
|
source: |
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=recogna-nlp/gembode-7b |
|
name: Open Portuguese LLM Leaderboard |
|
--- |
|
|
|
# gembode-7b |
|
|
|
<!--- PROJECT LOGO --> |
|
<p align="center"> |
|
<img src="https://huggingface.co/recogna-nlp/GemBode-2b-it/resolve/main/gembode.jpg" alt="Phi-Bode Logo" width="400" style="margin-left:'auto' margin-right:'auto' display:'block'"/> |
|
</p> |
|
|
|
GemmBode é um modelo de linguagem ajustado para o idioma português, desenvolvido a partir do modelo Gemma-7b fornecido pela [Google](https://huggingface.co/google/gemma-7b). |
|
|
|
## Características Principais |
|
|
|
- **Modelo Base:** Gemma-7b, criado pela Google, com 7 bilhões de parâmetros. |
|
- **Dataset para Fine-tuning:** [UltraAlpaca](https://huggingface.co/datasets/recogna-nlp/ultra-alpaca-ptbr) |
|
- **Treinamento:** O treinamento foi realizado a partir do fine-tuning, com QLoRA do gemma-7b. |
|
|
|
|
|
# Resultados da avaliação do Open Portuguese LLM Leaderboard |
|
|
|
Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/recogna-nlp/gembode-7b) and on the [🚀 Open Portuguese LLM Leaderboard](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard) |
|
|
|
| Metric | Value | |
|
|--------------------------|---------| |
|
|Average |**67.11**| |
|
|ENEM Challenge (No Images)| 66.90| |
|
|BLUEX (No Images) | 57.16| |
|
|OAB Exams | 45.47| |
|
|Assin2 RTE | 86.61| |
|
|Assin2 STS | 71.39| |
|
|FaQuAD NLI | 67.40| |
|
|HateBR Binary | 79.81| |
|
|PT Hate Speech Binary | 63.75| |
|
|tweetSentBR | 65.49| |
|
|
|
|