dennispark commited on
Commit
90c817c
β€’
1 Parent(s): a6cb06f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +41 -0
README.md CHANGED
@@ -1,3 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  | | Model | ynat (macro F1) | sts (pearsonr/F1) | nli (acc) | ner (entity-level F1) | re (micro F1) | dp (LAS) | mrc (EM/F1) |
2
  |-----|------------------------------------------------------------------|-----------------|-------------------|-----------|-----------------------|---------------|-----------|-------------|
3
  | | Baseline | **87.30** | **93.20/86.13** | **89.50** | 86.06 | 71.06 | 87.93 | **75.26/-** |
@@ -7,3 +42,9 @@
7
  | MT | pko-t5-small | 84.54 | 68.50/72/02 | 51.16 | 74.69 | 66.11 | 80.40 | 43.60/46.28 |
8
  | MT | pko-t5-base | 86.89 | 83.96/80.30 | 72.03 | 85.27 | 66.59 | 95.05 | 61.11/63.94 |
9
  | MT | pko-t5-large | 87.57 | 91.93/86.29 | 83.63 | 87.41 | 71.34 | 96.99 | 70.70/73.72 |
 
 
 
 
 
 
 
1
+ ---
2
+ language: ko
3
+
4
+ license: cc-by-4.0
5
+ ---
6
+
7
+ # pko-t5-base
8
+ [Source Code](https://github.com/paust-team/pko-t5)
9
+
10
+ pko-t5 λŠ” ν•œκ΅­μ–΄ μ „μš© λ°μ΄ν„°λ‘œ ν•™μŠ΅ν•œ [t5 v1.1 λͺ¨λΈ](https://github.com/google-research/text-to-text-transfer-transformer/blob/84f8bcc14b5f2c03de51bd3587609ba8f6bbd1cd/released_checkpoints.md)μž…λ‹ˆλ‹€.
11
+
12
+ ν•œκ΅­μ–΄λ₯Ό tokenize ν•˜κΈ° μœ„ν•΄μ„œ sentencepiece λŒ€μ‹  OOV κ°€ μ—†λŠ” BBPE λ₯Ό μ‚¬μš©ν–ˆμœΌλ©° ν•œκ΅­μ–΄ 데이터 (λ‚˜λ¬΄μœ„ν‚€, μœ„ν‚€ν”Όλ””μ•„, λͺ¨λ‘μ˜λ§λ­‰μΉ˜ λ“±..) λ₯Ό T5 의 span corruption task λ₯Ό μ‚¬μš©ν•΄μ„œ unsupervised learning 만 μ μš©ν•˜μ—¬ ν•™μŠ΅μ„ μ§„ν–‰ν–ˆμŠ΅λ‹ˆλ‹€.
13
+
14
+ pko-t5 λ₯Ό μ‚¬μš©ν•˜μ‹€ λ•ŒλŠ” λŒ€μƒ task 에 νŒŒμΈνŠœλ‹ν•˜μ—¬ μ‚¬μš©ν•˜μ‹œκΈ° λ°”λžλ‹ˆλ‹€.
15
+
16
+ ## Usage
17
+ transformers 의 API λ₯Ό μ‚¬μš©ν•˜μ—¬ μ ‘κ·Ό κ°€λŠ₯ν•©λ‹ˆλ‹€. tokenizer λ₯Ό μ‚¬μš©ν• λ•ŒλŠ” `T5Tokenizer` κ°€ μ•„λ‹ˆλΌ `T5TokenizerFast` λ₯Ό μ‚¬μš©ν•΄μ£Όμ‹­μ‹œμ˜€. model 은 T5ForConditionalGeneration λ₯Ό κ·ΈλŒ€λ‘œ ν™œμš©ν•˜μ‹œλ©΄ λ©λ‹ˆλ‹€.
18
+
19
+ ### Example
20
+ ```python
21
+ from transformers import T5TokenizerFast, T5ForConditionalGeneration
22
+
23
+ tokenizer = T5TokenizerFast.from_pretrained('paust/pko-t5-base')
24
+ model = T5ForConditionalGeneration.from_pretrained('paust/pko-t5-base')
25
+
26
+ input_ids = tokenizer(["qa question: λ‹Ήμ‹ μ˜ 이름은 λ¬΄μ—‡μΈκ°€μš”?"]).input_ids
27
+ labels = tokenizer(["T5 μž…λ‹ˆλ‹€."]).input_ids
28
+ outputs = model(input_ids=input_ids, labels=labels)
29
+
30
+ print(f"loss={outputs.loss} logits={outputs.logits}")
31
+ ```
32
+
33
+
34
+ ## Klue 평가 (dev)
35
+
36
  | | Model | ynat (macro F1) | sts (pearsonr/F1) | nli (acc) | ner (entity-level F1) | re (micro F1) | dp (LAS) | mrc (EM/F1) |
37
  |-----|------------------------------------------------------------------|-----------------|-------------------|-----------|-----------------------|---------------|-----------|-------------|
38
  | | Baseline | **87.30** | **93.20/86.13** | **89.50** | 86.06 | 71.06 | 87.93 | **75.26/-** |
 
42
  | MT | pko-t5-small | 84.54 | 68.50/72/02 | 51.16 | 74.69 | 66.11 | 80.40 | 43.60/46.28 |
43
  | MT | pko-t5-base | 86.89 | 83.96/80.30 | 72.03 | 85.27 | 66.59 | 95.05 | 61.11/63.94 |
44
  | MT | pko-t5-large | 87.57 | 91.93/86.29 | 83.63 | 87.41 | 71.34 | 96.99 | 70.70/73.72 |
45
+
46
+ - FT: μ‹±κΈ€νƒœμŠ€ν¬ νŒŒμΈνŠœλ‹ / MT: λ©€ν‹°νƒœμŠ€ν¬ νŒŒμΈνŠœλ‹
47
+ - [Baseline](https://arxiv.org/abs/2105.09680): KLUE λ…Όλ¬Έμ—μ„œ μ†Œκ°œλœ dev set 에 λŒ€ν•œ SOTA 점수
48
+
49
+ ## License
50
+ [PAUST](https://paust.io)μ—μ„œ λ§Œλ“  pko-t5λŠ” [MIT license](https://github.com/paust-team/pko-t5/blob/main/LICENSE) ν•˜μ— κ³΅κ°œλ˜μ–΄ μžˆμŠ΅λ‹ˆλ‹€.