|
--- |
|
model-index: |
|
- name: EEVE-Math-10.8B |
|
results: |
|
- task: |
|
type: text-generation |
|
dataset: |
|
name: gsm8k-ko |
|
type: gsm8k |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 0.539 |
|
verified: false |
|
base_model: yanolja/EEVE-Korean-10.8B-v1.0 |
|
license: cc-by-sa-4.0 |
|
language: |
|
- ko |
|
tags: |
|
- math |
|
datasets: |
|
- kuotient/orca-math-word-problems-193k-korean |
|
--- |
|
# EEVE-Math-10.8B |
|
|
|
`EEVE-Math` νλ‘μ νΈλ |
|
- Orca-Math-200k λ²μ ([Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://arxiv.org/pdf/2402.14830.pdf)) |
|
- gsm8k λ²μ, lm_eval νμ© |
|
- Mergekitμ μ΄μ©ν dare-ties μ¬μ© ([DARE](https://arxiv.org/abs/2311.03099)) |
|
|
|
μ λν λ΄μ©μ ν¬κ΄νκ³ μμ΅λλ€. |
|
|
|
> μ΄ λͺ¨λΈμ orca-math-word-problems-193k-korean λ°μ΄ν°μ
μ μ΄μ©νμ¬ νμ΅λμμ΅λλ€. μλ΅ μ€ μΌλΆλ LaTeX νμμ μ΄μ©νμ¬ κ²°κ³Όλ₯Ό λ°ννμ§λ§, μμ±λ νμμ΄ μλ μ μμ΅λλ€. νμ¬ M1 stageκΉμ§ μ§νλμμ΅λλ€. |
|
|
|
| Model | gsm8k-ko(pass@1) | |
|
|---|---| |
|
| Base | 0.4049 | |
|
| SFT(M1) | 0.508 | |
|
| SFT(M1) -> SFT | **0.539** | |
|
| SFT(M1) -> KTO(M2) | - | |
|
| SFT -> KTO(M2) -> KTO(final) | - | |
|
|
|
|
|
## Specifications |
|
- SFT(M1) -> SFT λ¨κ³ |
|
|
|
## Base Model |
|
[yanolja/EEVE-Korean-10.8B-v1.0](https://huggingface.co/yanolja/EEVE-Korean-10.8B-v1.0) |
|
|
|
## Dataset |
|
[orca-math-word-problems-193k-korean](https://huggingface.co/datasets/kuotient/orca-math-word-problems-193k-korean) |
|
|
|
## Evaluation |
|
[gsm8k-ko](https://huggingface.co/datasets/kuotient/gsm8k-ko), kobest |
|
``` |
|
git clone https://github.com/kuotient/lm-evaluation-harness.git |
|
cd lm-evaluation-harness |
|
pip install -e . |
|
``` |
|
``` |
|
lm_eval --model hf \ |
|
--model_args pretrained=yanolja/EEVE-Korean-Instruct-2.8B-v1.0 \ |
|
--tasks gsm8k-ko \ |
|
--device cuda:0 \ |
|
--batch_size auto:4 |
|
``` |
|
|
|
| Model | gsm8k(pass@1) | boolq(acc) | copa(acc) | hellaswag(acc) | Overall | |
|
|---|---|---|---|---|---| |
|
| yanolja/EEVE-Korean-10.8B-v1.0 | 0.4049 | - | - | - | - | - | |
|
| yanolja/EEVE-Korean-Instruct-10.8B-v1.0 | 0.4511 | **0.8668** | **0.7450** | 0.4940 | 0.6392 | |
|
| **EEVE-Math-10.8B** | **0.5390** | 0.8027 | 0.7260 | 0.4760 | 0.6359 | |
|
| [**EEVE-Instruct-Math-10.8B**](https://huggingface.co/kuotient/EEVE-Instruct-Math-10.8B) | 0.4845 | 0.8519 | 0.7410 | **0.4980** | **0.6439** | |
