Hugging Face's logo

kuotient
/
EEVE-Math-10.8B

Text Generation
Transformers
PyTorch
Korean
llama
math
conversational
Eval Results
Inference Endpoints
text-generation-inference
Model card Files Community
1
EEVE-Math-10.8B
File size: 2,299 Bytes 
7d9d4ae
 
34510d6
 
 
 
 
 
 
 
 
 
 
 
7d9d4ae
34510d6
 
 
 
 
d72cd53
 
7d9d4ae
6bfc805
fc8d748
73a59a2
 
 
 
 
 
 
 
7d9d4ae
2d36827
7d9d4ae
 
7d0486a
 
 
 
7d9d4ae
 
 
7d0486a
7d9d4ae
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f549b42
ccfff81
f549b42
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
---
model-index:
- name: EEVE-Math-10.8B
  results:
  - task:
      type: text-generation
    dataset:
      name: gsm8k-ko
      type: gsm8k
    metrics:
    - name: pass@1
      type: pass@1
      value: 0.539
      verified: false
base_model: yanolja/EEVE-Korean-10.8B-v1.0
license: cc-by-sa-4.0
language:
- ko
tags:
- math
datasets:
- kuotient/orca-math-word-problems-193k-korean
---
# EEVE-Math-10.8B

`EEVE-Math` ν”„λ‘œμ νŠΈλŠ”
- Orca-Math-200k λ²ˆμ—­ ([Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://arxiv.org/pdf/2402.14830.pdf))
- gsm8k λ²ˆμ—­, lm_eval ν™œμš©
- Mergekit을 μ΄μš©ν•œ dare-ties μ‚¬μš© ([DARE](https://arxiv.org/abs/2311.03099))

에 λŒ€ν•œ λ‚΄μš©μ„ ν¬κ΄„ν•˜κ³  μžˆμŠ΅λ‹ˆλ‹€.

> 이 λͺ¨λΈμ€ orca-math-word-problems-193k-korean 데이터셋을 μ΄μš©ν•˜μ—¬ ν•™μŠ΅λ˜μ—ˆμŠ΅λ‹ˆλ‹€. 응닡 쀑 μΌλΆ€λŠ” LaTeX ν˜•μ‹μ„ μ΄μš©ν•˜μ—¬ κ²°κ³Όλ₯Ό λ°˜ν™˜ν•˜μ§€λ§Œ, μ™„μ„±λœ ν˜•μ‹μ΄ 아닐 수 μžˆμŠ΅λ‹ˆλ‹€. ν˜„μž¬ M1 stageκΉŒμ§€ μ§„ν–‰λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

| Model | gsm8k-ko(pass@1) |
|---|---|
| Base | 0.4049 |
| SFT(M1) | 0.508 |
| SFT(M1) -> SFT | **0.539** |
| SFT(M1) -> KTO(M2) | - |
| SFT -> KTO(M2) -> KTO(final) | - |


## Specifications
- SFT(M1) -> SFT 단계

## Base Model
[yanolja/EEVE-Korean-10.8B-v1.0](https://huggingface.co/yanolja/EEVE-Korean-10.8B-v1.0)

## Dataset
[orca-math-word-problems-193k-korean](https://huggingface.co/datasets/kuotient/orca-math-word-problems-193k-korean)

## Evaluation
[gsm8k-ko](https://huggingface.co/datasets/kuotient/gsm8k-ko), kobest
```
git clone https://github.com/kuotient/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e .
```
```
lm_eval --model hf \
    --model_args pretrained=yanolja/EEVE-Korean-Instruct-2.8B-v1.0 \
    --tasks gsm8k-ko \
    --device cuda:0 \
    --batch_size auto:4
```

| Model | gsm8k(pass@1) | boolq(acc) | copa(acc) | hellaswag(acc) | Overall |
|---|---|---|---|---|---|
| yanolja/EEVE-Korean-10.8B-v1.0 | 0.4049 | - | - | - | - | - |
| yanolja/EEVE-Korean-Instruct-10.8B-v1.0 | 0.4511 | **0.8668** | **0.7450** | 0.4940 | 0.6392 |
| **EEVE-Math-10.8B** | **0.5390** | 0.8027 | 0.7260 | 0.4760 | 0.6359 |
| [**EEVE-Instruct-Math-10.8B**](https://huggingface.co/kuotient/EEVE-Instruct-Math-10.8B) | 0.4845 | 0.8519 | 0.7410 | **0.4980** | **0.6439** |