|
--- |
|
license: mit |
|
language: |
|
- ko |
|
pipeline_tag: fill-mask |
|
--- |
|
# KF-DeBERTa |
|
์นด์นด์ค๋ฑ
ํฌ & ์ํ์๊ฐ์ด๋์์ ํ์ตํ ๊ธ์ต ๋๋ฉ์ธ ํนํ ์ธ์ด๋ชจ๋ธ์ ๊ณต๊ฐํฉ๋๋ค. |
|
|
|
## Model description |
|
* KF-DeBERTa๋ ๋ฒ์ฉ ๋๋ฉ์ธ ๋ง๋ญ์น์ ๊ธ์ต ๋๋ฉ์ธ ๋ง๋ญ์น๋ฅผ ํจ๊ป ํ์ตํ ์ธ์ด๋ชจ๋ธ ์
๋๋ค. |
|
* ๋ชจ๋ธ ์ํคํ
์ณ๋ [DeBERTa-v2](https://github.com/microsoft/DeBERTa#whats-new-in-v2)๋ฅผ ๊ธฐ๋ฐ์ผ๋ก ํ์ตํ์์ต๋๋ค. |
|
* ELECTRA์ RTD๋ฅผ training objective๋ก ์ฌ์ฉํ DeBERTa-v3๋ ์ผ๋ถ task(KLUE-RE, WoS, Retrieval)์์ ์๋นํ ๋ฎ์ ์ฑ๋ฅ์ ํ์ธํ์ฌ ์ต์ข
์ํคํ
์ณ๋ DeBERTa-v2๋ก ๊ฒฐ์ ํ์์ต๋๋ค. |
|
* ๋ฒ์ฉ ๋๋ฉ์ธ ๋ฐ ๊ธ์ต ๋๋ฉ์ธ downstream task์์ ๋ชจ๋ ์ฐ์ํ ์ฑ๋ฅ์ ํ์ธํ์์ต๋๋ค. |
|
* ๊ธ์ต ๋๋ฉ์ธ downstream task์ ์ฒ ์ ํ ์ฑ๋ฅ๊ฒ์ฆ์ ์ํด ๋ค์ํ ๋ฐ์ดํฐ์
์ ํตํด ๊ฒ์ฆ์ ์ํํ์์ต๋๋ค. |
|
* ๋ฒ์ฉ ๋๋ฉ์ธ ๋ฐ ๊ธ์ต ๋๋ฉ์ธ์์ ๊ธฐ์กด ์ธ์ด๋ชจ๋ธ๋ณด๋ค ๋ ๋์ ์ฑ๋ฅ์ ๋ณด์ฌ์คฌ์ผ๋ฉฐ ํนํ KLUE Benchmark์์๋ RoBERTa-Large๋ณด๋ค ๋ ๋์ ์ฑ๋ฅ์ ํ์ธํ์์ต๋๋ค. |
|
|
|
## Usage |
|
```python3 |
|
from transformers import AutoModel, AutoTokenizer |
|
|
|
model = AutoModel.from_pretrained("kakaobank/kf-deberta-base") |
|
tokenizer = AutoTokenizer.from_pretrained("kakaobank/kf-deberta-base") |
|
|
|
text = "์นด์นด์ค๋ฑ
ํฌ์ ์ํ์๊ฐ์ด๋๊ฐ ๊ธ์ตํนํ ์ธ์ด๋ชจ๋ธ์ ๊ณต๊ฐํฉ๋๋ค." |
|
tokens = tokenizer.tokenize(text) |
|
print(tokens) |
|
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
model_output = model(**inputs) |
|
print(model_output) |
|
``` |
|
|
|
## Benchmark |
|
* ๋ชจ๋ task๋ ์๋์ ๊ฐ์ ๊ธฐ๋ณธ์ ์ธ hyperparameter search๋ง ์ํํ์์ต๋๋ค. |
|
* batch size: {16, 32} |
|
* learning_rate: {1e-5, 3e-5, 5e-5} |
|
* weight_decay: {0, 0.01} |
|
* warmup_proportion: {0, 0.1} |
|
|
|
**KLUE Benchmark** |
|
|
|
| Model | YNAT | KLUE-ST | KLUE-NLI | KLUE-NER | KLUE-RE | KLUE-DP | KLUE-MRC | WoS | AVG | |
|
|:--------------------:|:----------------:|:----------------------:|:------------:|:---------------------------------:|:-----------------------------:|:----------------------:|:-------------------------:|:----------------------:|:----------------:| |
|
| | F1 | Pearsonr/F1 | ACC | F1-Entity/F1-Char | F1-micro/AUC | UAS/LAS | EM/ROUGE | JGA/F1-S | | |
|
| mBERT (Base) | 82.64 | 82.97/75.93 | 72.90 | 75.56/88.81 | 58.39/56.41 | 88.53/86.04 | 49.96/55.57 | 35.27/88.60 | 71.26 | |
|
| XLM-R (Base) | 84.52 | 88.88/81.20 | 78.23 | 80.48/92.14 | 57.62/57.05 | 93.12/87.23 | 26.76/53.36 | 41.54/89.81 | 72.28 | |
|
| XLM-R (Large) | 87.30 | 93.08/87.17 | 86.40 | 82.18/93.20 | 58.75/63.53 | 92.87/87.82 | 35.23/66.55 | 42.44/89.88 | 76.17 | |
|
| KR-BERT (Base) | 85.36 | 87.50/77.92 | 77.10 | 74.97/90.46 | 62.83/65.42 | 92.87/87.13 | 48.95/58.38 | 45.60/90.82 | 74.67 | |
|
| KoELECTRA (Base) | 85.99 | 93.14/85.89 | 86.87 | 86.06/92.75 | 62.67/57.46 | 90.93/87.07 | 59.54/65.64 | 39.83/88.91 | 77.34 | |
|
| KLUE-BERT (Base) | 86.95 | 91.01/83.44 | 79.87 | 83.71/91.17 | 65.58/68.11 | 93.07/87.25 | 62.42/68.15 | 46.72/91.59 | 78.50 | |
|
| KLUE-RoBERTa (Small) | 85.95 | 91.70/85.42 | 81.00 | 83.55/91.20 | 61.26/60.89 | 93.47/87.50 | 58.25/63.56 | 46.65/91.50 | 77.28 | |
|
| KLUE-RoBERTa (Base) | 86.19 | 92.91/86.78 | 86.30 | 83.81/91.09 | 66.73/68.11 | 93.75/87.77 | 69.56/74.64 | 47.41/91.60 | 80.48 | |
|
| KLUE-RoBERTa (Large) | 85.88 | 93.20/86.13 | **89.50** | 84.54/91.45 | **71.06**/73.33 | 93.84/87.93 | **75.26**/**80.30** | 49.39/92.19 | 82.43 | |
|
| KF-DeBERTa (Base) | **<u>87.51</u>** | **<u>93.24/87.73</u>** | <u>88.37</u> | **<u>89.17</u>**/**<u>93.30</u>** | <u>69.70</u>/**<u>75.07</u>** | **<u>94.05/87.97</u>** | <u>72.59</u>/<u>78.08</u> | **<u>50.21/92.59</u>** | **<u>82.83</u>** | |
|
|
|
* ๊ตต์๊ธ์จ๋ ๋ชจ๋ ๋ชจ๋ธ์ค ๊ฐ์ฅ๋์ ์ ์์ด๋ฉฐ, ๋ฐ์ค์ base ๋ชจ๋ธ ์ค ๊ฐ์ฅ ๋์ ์ ์์
๋๋ค. |
|
|
|
**๊ธ์ต๋๋ฉ์ธ ๋ฒค์น๋งํฌ** |
|
| Model | FN-Sentiment (v1) | FN-Sentiment (v2) | FN-Adnews | FN-NER | KorFPB | KorFiQA-SA | KorHeadline | Avg (FiQA-SA ์ ์ธ) | |
|
|:-------------------:|:-----------------:|:-----------------:|:---------:|:---------:|:---------:|:----------:|:-----------:|:-----------------:| |
|
| | ACC | ACC | ACC | F1-micro | ACC | MSE | Mean F1 | | |
|
| KLUE-RoBERTa (Base) | 98.26 | 91.21 | 96.34 | 90.31 | 90.97 | 0.0589 | 81.11 | 94.03 | |
|
| KoELECTRA (Base) | 98.26 | 90.56 | 96.98 | 89.81 | 92.36 | 0.0652 | 80.69 | 93.90 | |
|
| KF-DeBERTa (Base) | **99.36** | **92.29** | **97.63** | **91.80** | **93.47** | **0.0553** | **82.12** | **95.27** | |
|
|
|
* **FN-Sentiment**: ๊ธ์ต๋๋ฉ์ธ ๊ฐ์ฑ๋ถ์ |
|
* **FN-Adnews**: ๊ธ์ต๋๋ฉ์ธ ๊ด๊ณ ์ฑ๊ธฐ์ฌ ๋ถ๋ฅ |
|
* **FN-NER**: ๊ธ์ต๋๋ฉ์ธ ๊ฐ์ฒด๋ช
์ธ์ |
|
* **KorFPB**: FinancialPhraseBank ๋ฒ์ญ๋ฐ์ดํฐ |
|
* Cite: ```Malo, Pekka, et al. "Good debt or bad debt: Detecting semantic orientations in economic texts." Journal of the Association for Information Science and Technology 65.4 (2014): 782-796.``` |
|
* **KorFiQA-SA**: FiQA-SA ๋ฒ์ญ๋ฐ์ดํฐ |
|
* Cite: ```Maia, Macedo & Handschuh, Siegfried & Freitas, Andre & Davis, Brian & McDermott, Ross & Zarrouk, Manel & Balahur, Alexandra. (2018). WWW'18 Open Challenge: Financial Opinion Mining and Question Answering. WWW '18: Companion Proceedings of the The Web Conference 2018. 1941-1942. 10.1145/3184558.3192301.``` |
|
* **KorHeadline**: Gold Commodity News and Dimensions ๋ฒ์ญ๋ฐ์ดํฐ |
|
* Cite: ```Sinha, A., & Khandait, T. (2021, April). Impact of News on the Commodity Market: Dataset and Results. In |
|
Future of Information and Communication Conference (pp. 589-601). Springer, Cham.``` |
|
|
|
|
|
**๋ฒ์ฉ๋๋ฉ์ธ ๋ฒค์น๋งํฌ** |
|
| Model | NSMC | PAWS | KorNLI | KorSTS | KorQuAD | Avg (KorQuAD ์ ์ธ) | |
|
|:-------------------:|:---------:|:---------:|:---------:|:---------:|:---------------:|:----------------:| |
|
| | ACC | ACC | ACC | spearman | EM/F1 | | |
|
| KLUE-RoBERTa (Base) | 90.47 | 84.79 | 81.65 | 84.40 | 86.34/94.40 | 85.33 | |
|
| KoELECTRA (Base) | 90.63 | 84.45 | 82.24 | 85.53 | 84.83/93.45 | 85.71 | |
|
| KF-DeBERTa (Base) | **91.36** | **86.14** | **84.54** | **85.99** | **86.60/95.07** | **87.01** | |
|
|
|
|
|
|
|
## License |
|
KF-DeBERTa์ ์์ค์ฝ๋ ๋ฐ ๋ชจ๋ธ์ MIT ๋ผ์ด์ ์ค ํ์ ๊ณต๊ฐ๋์ด ์์ต๋๋ค. |
|
๋ผ์ด์ ์ค ์ ๋ฌธ์ [MIT ํ์ผ](LICENSE)์์ ํ์ธํ ์ ์์ต๋๋ค. |
|
๋ชจ๋ธ์ ์ฌ์ฉ์ผ๋ก ์ธํด ๋ฐ์ํ ์ด๋ ํ ์ํด์ ๋ํด์๋ ๋น์ฌ๋ ์ฑ
์์ ์ง์ง ์์ต๋๋ค. |
|
|
|
## Citation |
|
``` |
|
@proceedings{jeon-etal-2023-kfdeberta, |
|
title = {KF-DeBERTa: Financial Domain-specific Pre-trained Language Model}, |
|
author = {Eunkwang Jeon, Jungdae Kim, Minsang Song, and Joohyun Ryu}, |
|
booktitle = {Proceedings of the 35th Annual Conference on Human and Cognitive Language Technology}, |
|
moth = {oct}, |
|
year = {2023}, |
|
publisher = {Korean Institute of Information Scientists and Engineers}, |
|
url = {http://www.hclt.kr/symp/?lnb=conference}, |
|
pages = {143--148}, |
|
} |
|
``` |
