dennispark
commited on
Commit
β’
90c817c
1
Parent(s):
a6cb06f
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
| | Model | ynat (macro F1) | sts (pearsonr/F1) | nli (acc) | ner (entity-level F1) | re (micro F1) | dp (LAS) | mrc (EM/F1) |
|
2 |
|-----|------------------------------------------------------------------|-----------------|-------------------|-----------|-----------------------|---------------|-----------|-------------|
|
3 |
| | Baseline | **87.30** | **93.20/86.13** | **89.50** | 86.06 | 71.06 | 87.93 | **75.26/-** |
|
@@ -7,3 +42,9 @@
|
|
7 |
| MT | pko-t5-small | 84.54 | 68.50/72/02 | 51.16 | 74.69 | 66.11 | 80.40 | 43.60/46.28 |
|
8 |
| MT | pko-t5-base | 86.89 | 83.96/80.30 | 72.03 | 85.27 | 66.59 | 95.05 | 61.11/63.94 |
|
9 |
| MT | pko-t5-large | 87.57 | 91.93/86.29 | 83.63 | 87.41 | 71.34 | 96.99 | 70.70/73.72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language: ko
|
3 |
+
|
4 |
+
license: cc-by-4.0
|
5 |
+
---
|
6 |
+
|
7 |
+
# pko-t5-base
|
8 |
+
[Source Code](https://github.com/paust-team/pko-t5)
|
9 |
+
|
10 |
+
pko-t5 λ νκ΅μ΄ μ μ© λ°μ΄ν°λ‘ νμ΅ν [t5 v1.1 λͺ¨λΈ](https://github.com/google-research/text-to-text-transfer-transformer/blob/84f8bcc14b5f2c03de51bd3587609ba8f6bbd1cd/released_checkpoints.md)μ
λλ€.
|
11 |
+
|
12 |
+
νκ΅μ΄λ₯Ό tokenize νκΈ° μν΄μ sentencepiece λμ OOV κ° μλ BBPE λ₯Ό μ¬μ©νμΌλ©° νκ΅μ΄ λ°μ΄ν° (λ무μν€, μν€νΌλμ, λͺ¨λμλ§λμΉ λ±..) λ₯Ό T5 μ span corruption task λ₯Ό μ¬μ©ν΄μ unsupervised learning λ§ μ μ©νμ¬ νμ΅μ μ§ννμ΅λλ€.
|
13 |
+
|
14 |
+
pko-t5 λ₯Ό μ¬μ©νμ€ λλ λμ task μ νμΈνλνμ¬ μ¬μ©νμκΈ° λ°λλλ€.
|
15 |
+
|
16 |
+
## Usage
|
17 |
+
transformers μ API λ₯Ό μ¬μ©νμ¬ μ κ·Ό κ°λ₯ν©λλ€. tokenizer λ₯Ό μ¬μ©ν λλ `T5Tokenizer` κ° μλλΌ `T5TokenizerFast` λ₯Ό μ¬μ©ν΄μ£Όμμμ€. model μ T5ForConditionalGeneration λ₯Ό κ·Έλλ‘ νμ©νμλ©΄ λ©λλ€.
|
18 |
+
|
19 |
+
### Example
|
20 |
+
```python
|
21 |
+
from transformers import T5TokenizerFast, T5ForConditionalGeneration
|
22 |
+
|
23 |
+
tokenizer = T5TokenizerFast.from_pretrained('paust/pko-t5-base')
|
24 |
+
model = T5ForConditionalGeneration.from_pretrained('paust/pko-t5-base')
|
25 |
+
|
26 |
+
input_ids = tokenizer(["qa question: λΉμ μ μ΄λ¦μ 무μμΈκ°μ?"]).input_ids
|
27 |
+
labels = tokenizer(["T5 μ
λλ€."]).input_ids
|
28 |
+
outputs = model(input_ids=input_ids, labels=labels)
|
29 |
+
|
30 |
+
print(f"loss={outputs.loss} logits={outputs.logits}")
|
31 |
+
```
|
32 |
+
|
33 |
+
|
34 |
+
## Klue νκ° (dev)
|
35 |
+
|
36 |
| | Model | ynat (macro F1) | sts (pearsonr/F1) | nli (acc) | ner (entity-level F1) | re (micro F1) | dp (LAS) | mrc (EM/F1) |
|
37 |
|-----|------------------------------------------------------------------|-----------------|-------------------|-----------|-----------------------|---------------|-----------|-------------|
|
38 |
| | Baseline | **87.30** | **93.20/86.13** | **89.50** | 86.06 | 71.06 | 87.93 | **75.26/-** |
|
|
|
42 |
| MT | pko-t5-small | 84.54 | 68.50/72/02 | 51.16 | 74.69 | 66.11 | 80.40 | 43.60/46.28 |
|
43 |
| MT | pko-t5-base | 86.89 | 83.96/80.30 | 72.03 | 85.27 | 66.59 | 95.05 | 61.11/63.94 |
|
44 |
| MT | pko-t5-large | 87.57 | 91.93/86.29 | 83.63 | 87.41 | 71.34 | 96.99 | 70.70/73.72 |
|
45 |
+
|
46 |
+
- FT: μ±κΈνμ€ν¬ νμΈνλ / MT: λ©ν°νμ€ν¬ νμΈνλ
|
47 |
+
- [Baseline](https://arxiv.org/abs/2105.09680): KLUE λ
Όλ¬Έμμ μκ°λ dev set μ λν SOTA μ μ
|
48 |
+
|
49 |
+
## License
|
50 |
+
[PAUST](https://paust.io)μμ λ§λ pko-t5λ [MIT license](https://github.com/paust-team/pko-t5/blob/main/LICENSE) νμ 곡κ°λμ΄ μμ΅λλ€.
|