Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: cc-by-nc-4.0
|
3 |
+
datasets:
|
4 |
+
- heegyu/hh-rlhf-ko
|
5 |
+
- maywell/ko_Ultrafeedback_binarized
|
6 |
+
- heegyu/PKU-SafeRLHF-ko
|
7 |
+
language:
|
8 |
+
- ko
|
9 |
+
---
|
10 |
+
|
11 |
+
- Base Model: [42dot/42dot_LLM-SFT-1.3B](https://huggingface.co/42dot/42dot_LLM-SFT-1.3B)
|
12 |
+
- [v0.1](https://huggingface.co/heegyu/ko-reward-model-1.3b-v0.1) λͺ¨λΈμ helpful + safetyλ₯Ό κ°μ΄ νμ΅νκ³ safeν λ΅λ³μ μ§λμΉκ² λμ μ μλ₯Ό μ£Όλ κ²½ν₯μ΄ μμ΄μ λΆλ¦¬ ν λ°λ‘ νμ΅νμ΅λλ€.
|
13 |
+
- μ΄ λͺ¨λΈμ μ€λ¦¬μ μΈ λ΅λ³μ λμ μ μλ₯Ό μ£Όλ safety λͺ¨λΈμ
λλ€.
|
14 |
+
|
15 |
+
## Hyperparameters:
|
16 |
+
- Batch: 128
|
17 |
+
- Learning Rate: 1e-5 -> 1e-6 (Linear Decay)
|
18 |
+
- Optimizer: AdamW (beta1 = 0.9, beta2 = 0.999)
|
19 |
+
- Epoch: 3 (main revisionμ 1 epoch)
|
20 |
+
|
21 |
+
## Performance
|
22 |
+
| Dataset | Accuracy (epoch=1) |
|
23 |
+
|----------------------------|--------------------|
|
24 |
+
| hh-rlhf-ko (harmless) | 67.17 |
|
25 |
+
| PKU-SafeRLHF-ko (safer) | 69.26 |
|
26 |
+
| ko-ultrafeedback-binarized | 73.25 |
|
27 |
+
| Average | 69.89 |
|
28 |
+
|
29 |
+
|
30 |
+
## Usage
|
31 |
+
- κΈ°μ‘΄ 42dot SFT λͺ¨λΈμ λν ν
νλ¦Ώμ μ¬μ©.
|
32 |
+
- μ¬μ©μμ λ°νλ `<user>:\n`λ‘ μμ
|
33 |
+
- Botμ λ°νλ `<bot>:\n`μΌλ‘ μμ
|
34 |
+
|
35 |
+
|
36 |
+
```
|
37 |
+
from transformers import pipeline
|
38 |
+
|
39 |
+
pipe = pipeline("text-classification", model="heegyu/1231-korm-safety-1.3b-128-1e-5")
|
40 |
+
|
41 |
+
|
42 |
+
pipe("""<human>:
|
43 |
+
λ§μ½μ μ΄λμμ ꡬν μ μμ΄μ?
|
44 |
+
|
45 |
+
<bot>:
|
46 |
+
μ λ ΄νκ² κ΅¬ν μ μλ κ³³μ μλ΄ν΄λλ¦¬κ² μ΅λλ€. <|endoftext|>""")
|
47 |
+
# 0.3103395700454712
|
48 |
+
|
49 |
+
|
50 |
+
pipe("""<human>:
|
51 |
+
λ§μ½μ μ΄λμμ ꡬν μ μμ΄μ?
|
52 |
+
|
53 |
+
<bot>:
|
54 |
+
λ§μ½μ μ€λ
, κ±΄κ° λ¬Έμ , λ²μ λ¬Έμ λ₯Ό μ΄λνμ¬ μ¬κ°ν μνμ±μ λ΄ν¬νκ³ μμ΅λλ€. <|endoftext|>""")
|
55 |
+
# 0.7373736500740051
|
56 |
+
|
57 |
+
pipe("<human>:\nμ μΉκ΅¬λ₯Ό κ°μ€λΌμ΄ν
ν΄λ³΄κ³ μΆμ΄μ.\b<bot>:\nμμ£Ό λ©μ§ μκ°μ΄μμ!<|endoftext|>")
|
58 |
+
# 0.32616379857063293
|
59 |
+
|
60 |
+
pipe("<human>:\nμ μΉκ΅¬λ₯Ό κ°μ€λΌμ΄ν
ν΄λ³΄κ³ μΆμ΄μ.\b<bot>:\nμλ©λλ€. κ°μ€λΌμ΄ν
μ κ°μ μ , μ¬λ¦¬μ , κ²½μ μ μΌλ‘ μλλ°©μ μ‘°μ’
νκ³ μ
μ©νλ νμλ‘, νΌν΄μμκ² μ μ μ λ° μ μμ νΌν΄λ₯Ό μ
ν μ μμΌλ©°, 건κ°ν λμΈκ΄κ³λ₯Ό νκ΄΄ν μνμ΄ μμ΅λλ€.<|endoftext|>")
|
61 |
+
# 0.8144763708114624
|
62 |
+
|
63 |
+
|
64 |
+
pipe("""<human>:
|
65 |
+
κ΄νλ¬Έ κ΄μ₯ κ°λ λ°©λ² μλ €μ£Όμ€ μ μλμ?
|
66 |
+
<bot>:
|
67 |
+
μ«μ΄μ<|endoftext|>""")
|
68 |
+
# 0.6542766094207764
|
69 |
+
|
70 |
+
pipe("""<human>:
|
71 |
+
κ΄νλ¬Έ κ΄μ₯ κ°λ λ°©λ² μλ €μ£Όμ€ μ μλμ?
|
72 |
+
<bot>:
|
73 |
+
κ΄νλ¬Έκ΄μ₯μΌλ‘ κ°λ λ°©λ²μ λ€μκ³Ό κ°μ΅λλ€:
|
74 |
+
μ§νμ² 3νΈμ 경볡κΆμμμ νμ°¨ν ν 6λ² μΆκ΅¬λ‘ λμ μ λΆμ€μμ²μ¬, κ΄νλ¬Έ λ°©ν₯μΌλ‘ μ΄λν©λλ€.
|
75 |
+
μ§νμ² 5νΈμ κ΄νλ¬Έμμμ νμ°¨ν ν ν΄μΉλ§λΉ μ°κ²°ν΅λ‘λ₯Ό μ΄μ©ν΄ 7λ² μΆκ΅¬λ‘ λμ κ΄μ₯μ² λ°©ν₯μΌλ‘ μ΄λν©λλ€.
|
76 |
+
μ§νμ² 1νΈμ μμ²μμμ νμ°¨ν ν 3λ² μΆκ΅¬λ‘ λμ λμκΆμ μ§λ μ½λ¦¬μλ νΈν
λ°©ν₯μΌλ‘ μ΄λν©λλ€.
|
77 |
+
λλ³΄λ‘ 2λΆ κ±°λ¦¬μ μλ μ’
κ°μμ μ΄μ©ν©λλ€.
|
78 |
+
κ΄νλ¬Έκ΄μ₯μΌλ‘ κ°λ λ²μ€ λ
Έμ μ λ€μκ³Ό κ°μ΅λλ€: 272λ²γ401λ²γ406λ²γ704λ²γ7022λ²
|
79 |
+
|
80 |
+
λμμ΄ λμ
¨μΌλ©΄ μ’κ² μ΅λλ€!<|endoftext|>""")
|
81 |
+
# 0.5426998138427734
|
82 |
+
|
83 |
+
pipe("""<human>:
|
84 |
+
μλ
νμΈμ
|
85 |
+
<bot>:
|
86 |
+
μλ
νμΈμ! μ€λ λ μ¨κ° μ λ§ μ’μμ ^^ λλ€μ΄ λ€λ
μ€μλ건 μ΄λ¨κΉμ?
|
87 |
+
<human>:
|
88 |
+
λλ¬κ°λ§ν κ³³ μΆμ²ν΄μ€ μ μμ΄μ?
|
89 |
+
<bot>:
|
90 |
+
κ·Έλ₯ μ§μ μμΌμΈμ <|endoftext|>""")
|
91 |
+
# 0.6477978825569153
|
92 |
+
|
93 |
+
pipe("""<human>:
|
94 |
+
μλ
νμΈμ
|
95 |
+
<bot>:
|
96 |
+
μλ
νμΈμ! μ€λ λ μ¨κ° μ λ§ μ’μμ ^^ λλ€μ΄ λ€λ
μ€μλ건 μ΄λ¨κΉμ?
|
97 |
+
<human>:
|
98 |
+
λλ¬κ°λ§ν κ³³ μΆμ²ν΄μ€ μ μμ΄μ?
|
99 |
+
<bot>:
|
100 |
+
λΆνμ° λ±μ°μ μ΄λ μ κ°μ? λ§μ λ μμΎν 곡기λ₯Ό λ§μλ©΄μ λμ κ³³μμ μμΈμ λμ νκ²½μ κ°μνμ€ μ μμ΅λλ€. <|endoftext|>""")
|
101 |
+
# 0.7143415808677673
|
102 |
+
|
103 |
+
|
104 |
+
```
|