shwan commited on
Commit
1e48ca1
1 Parent(s): b4b4856

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +101 -0
README.md ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # KoMiniLM
2
+ 🐣 Korean mini language model
3
+
4
+ ## Overview
5
+ Current language models usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and online serving in real-life applications due to latency and capacity constraints. In this project, we release a light weight korean language model to address the aforementioned shortcomings of existing language models.
6
+
7
+ ## Quick tour
8
+ ```python
9
+ from transformers import AutoTokenizer, AutoModel
10
+
11
+ tokenizer = AutoTokenizer.from_pretrained("BM-K/KoMiniLM") # 23M model
12
+ model = AutoModel.from_pretrained("BM-K/KoMiniLM")
13
+
14
+ inputs = tokenizer("안녕 세상아!", return_tensors="pt")
15
+ outputs = model(**inputs)
16
+ ```
17
+
18
+ ## Update history
19
+ ** Updates on 2022.06.20 **
20
+ - Release KoMiniLM-bert-68M
21
+
22
+ ** Updates on 2022.05.24 **
23
+ - Release KoMiniLM-bert-23M
24
+
25
+ ## Pre-training
26
+ `Teacher Model`: [KLUE-BERT(base)](https://github.com/KLUE-benchmark/KLUE)
27
+
28
+ ### Object
29
+ Self-Attention Distribution and Self-Attention Value-Relation [[Wang et al., 2020]] were distilled from each discrete layer of the teacher model to the student model. Wang et al. distilled in the last layer of the transformer, but that was not the case in this project.
30
+
31
+ ### Data sets
32
+ |Data|News comments|News article|
33
+ |:----:|:----:|:----:|
34
+ |size|10G|10G|
35
+
36
+ ### Config
37
+ - **KoMiniLM-23M**
38
+ ```json
39
+ {
40
+ "architectures": [
41
+ "BartForPreTraining"
42
+ ],
43
+ "attention_probs_dropout_prob": 0.1,
44
+ "classifier_dropout": null,
45
+ "hidden_act": "gelu",
46
+ "hidden_dropout_prob": 0.1,
47
+ "hidden_size": 384,
48
+ "initializer_range": 0.02,
49
+ "intermediate_size": 1536,
50
+ "layer_norm_eps": 1e-12,
51
+ "max_position_embeddings": 512,
52
+ "model_type": "bart",
53
+ "num_attention_heads": 12,
54
+ "num_hidden_layers": 6,
55
+ "output_attentions": true,
56
+ "pad_token_id": 0,
57
+ "position_embedding_type": "absolute",
58
+ "return_dict": false,
59
+ "torch_dtype": "float32",
60
+ "transformers_version": "4.13.0",
61
+ "type_vocab_size": 2,
62
+ "use_cache": true,
63
+ "vocab_size": 32000
64
+ }
65
+ ```
66
+
67
+ ### Performance on subtasks
68
+ - The results of our fine-tuning experiments are an average of 3 runs for each task.
69
+ ```
70
+ cd KoMiniLM-Finetune
71
+ bash scripts/run_all_kominilm.sh
72
+ ```
73
+
74
+ || #Param | Average | NSMC<br>(Acc) | Naver NER<br>(F1) | PAWS<br>(Acc) | KorNLI<br>(Acc) | KorSTS<br>(Spearman) | Question Pair<br>(Acc) | KorQuaD<br>(Dev)<br>(EM/F1) |
75
+ |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
76
+ |KoBERT(KLUE)| 110M | 86.84 | 90.20±0.07 | 87.11±0.05 | 81.36±0.21 | 81.06±0.33 | 82.47±0.14 | 95.03±0.44 | 84.43±0.18 / <br>93.05±0.04 |
77
+ |KcBERT| 108M | 78.94 | 89.60±0.10 | 84.34±0.13 | 67.02±0.42| 74.17±0.52 | 76.57±0.51 | 93.97±0.27 | 60.87±0.27 / <br>85.01±0.14 |
78
+ |KoBERT(SKT)| 92M | 79.73 | 89.28±0.42 | 87.54±0.04 | 80.93±0.91 | 78.18±0.45 | 75.98±2.81 | 94.37±0.31 | 51.94±0.60 / <br>79.69±0.66 |
79
+ |DistilKoBERT| 28M | 74.73 | 88.39±0.08 | 84.22±0.01 | 61.74±0.45 | 70.22±0.14 | 72.11±0.27 | 92.65±0.16 | 52.52±0.48 / <br>76.00±0.71 |
80
+ | | | | | | | | | |
81
+ |**KoMiniLM<sup>†</sup>**| **68M** | 85.90 | 89.84±0.02 | 85.98±0.09 | 80.78±0.30 | 79.28±0.17 | 81.00±0.07 | 94.89±0.37 | 83.27±0.08 / <br>92.08±0.06 |
82
+ |**KoMiniLM<sup>†</sup>**| **23M** | 84.79 | 89.67±0.03 | 84.79±0.09 | 78.67±0.45 | 78.10±0.07 | 78.90±0.11 | 94.81±0.12 | 82.11±0.42 / <br>91.21±0.29 |
83
+
84
+ - [NSMC](https://github.com/e9t/nsmc) (Naver Sentiment Movie Corpus)
85
+ - [Naver NER](https://github.com/naver/nlp-challenge) (NER task on Naver NLP Challenge 2018)
86
+ - [PAWS](https://github.com/google-research-datasets/paws) (Korean Paraphrase Adversaries from Word Scrambling)
87
+ - [KorNLI/KorSTS](https://github.com/kakaobrain/KorNLUDatasets) (Korean Natural Language Understanding)
88
+ - [Question Pair](https://github.com/songys/Question_pair) (Paired Question)
89
+ - [KorQuAD](https://korquad.github.io/) (The Korean Question Answering Dataset)
90
+
91
+ <img src = "https://user-images.githubusercontent.com/55969260/174229747-279122dc-9d27-4da9-a6e7-f9f1fe1651f7.png"> <br>
92
+
93
+ ### User Contributed Examples
94
+ -
95
+
96
+ ## Reference
97
+ - [KLUE BERT](https://github.com/KLUE-benchmark/KLUE)
98
+ - [KcBERT](https://github.com/Beomi/KcBERT)
99
+ - [SKT KoBERT](https://github.com/SKTBrain/KoBERT)
100
+ - [DistilKoBERT](https://github.com/monologg/DistilKoBERT)
101
+ - [lassl](https://github.com/lassl/lassl)