Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Dialog-KoELECTRA
|
| 2 |
+
|
| 3 |
+
Github : [https://github.com/skplanetml/Dialog-KoELECTRA](https://github.com/skplanetml/Dialog-KoELECTRA)
|
| 4 |
+
|
| 5 |
+
## Introduction
|
| 6 |
+
|
| 7 |
+
**Dialog-KoELECTRA** is a language model specialized for dialogue. It was trained with 22GB colloquial and written style Korean text data. Dialog-ELECTRA model is made based on the [ELECTRA](https://openreview.net/pdf?id=r1xMH1BtvB) model. ELECTRA is a method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU.
|
| 8 |
+
|
| 9 |
+
<br>
|
| 10 |
+
|
| 11 |
+
## Released Models
|
| 12 |
+
|
| 13 |
+
We are initially releasing small version pre-trained model.
|
| 14 |
+
The model was trained on Korean text. We hope to release other models, such as base/large models, in the future.
|
| 15 |
+
|
| 16 |
+
| Model | Layers | Hidden Size | Params | Max<br/>Seq Len | Learning<br/>Rate | Batch Size | Train Steps |
|
| 17 |
+
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 18 |
+
| Dialog-KoELECTRA-Small | 12 | 256 | 14M | 128 | 1e-4 | 512 | 700K |
|
| 19 |
+
|
| 20 |
+
<br>
|
| 21 |
+
|
| 22 |
+
## Model Performance
|
| 23 |
+
|
| 24 |
+
Dialog-KoELECTRA shows strong performance in conversational downstream tasks.
|
| 25 |
+
|
| 26 |
+
| | **NSMC**<br/>(acc) | **Question Pair**<br/>(acc) | **Korean-Hate-Speech**<br/>(F1) | **Naver NER**<br/>(F1) | **KorNLI**<br/>(acc) | **KorSTS**<br/>(spearman) |
|
| 27 |
+
| :--------------------- | :----------------: | :--------------------: | :----------------: | :------------------: | :-----------------------: | :-------------------------: |
|
| 28 |
+
| DistilKoBERT | 88.60 | 92.48 | 60.72 | 84.65 | 72.00 | 72.59 |
|
| 29 |
+
| **Dialog-KoELECTRA-Small** | **90.01** | **94.99** | **68.26** | **85.51** | **78.54** | **78.96** |
|
| 30 |
+
|
| 31 |
+
<br>
|
| 32 |
+
|
| 33 |
+
## Train Data
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
<table class="tg">
|
| 37 |
+
<thead>
|
| 38 |
+
<tr>
|
| 39 |
+
<th class="tg-c3ow"></th>
|
| 40 |
+
<th class="tg-c3ow">corpus name</th>
|
| 41 |
+
<th class="tg-c3ow">size</th>
|
| 42 |
+
</tr>
|
| 43 |
+
</thead>
|
| 44 |
+
<tbody>
|
| 45 |
+
<tr>
|
| 46 |
+
<td class="tg-c3ow" rowspan="4">dialog</td>
|
| 47 |
+
<td class="tg-0pky"><a href="https://aihub.or.kr/aidata/85" target="_blank" rel="noopener noreferrer">Aihub Korean dialog corpus</a></td>
|
| 48 |
+
<td class="tg-c3ow" rowspan="4">7GB</td>
|
| 49 |
+
</tr>
|
| 50 |
+
<tr>
|
| 51 |
+
<td class="tg-0pky"><a href="https://corpus.korean.go.kr/" target="_blank" rel="noopener noreferrer">NIKL Spoken corpus</a></td>
|
| 52 |
+
</tr>
|
| 53 |
+
<tr>
|
| 54 |
+
<td class="tg-0pky"><a href="https://github.com/songys/Chatbot_data" target="_blank" rel="noopener noreferrer">Korean chatbot data</a></td>
|
| 55 |
+
</tr>
|
| 56 |
+
<tr>
|
| 57 |
+
<td class="tg-0pky"><a href="https://github.com/Beomi/KcBERT" target="_blank" rel="noopener noreferrer">KcBERT</a></td>
|
| 58 |
+
</tr>
|
| 59 |
+
<tr>
|
| 60 |
+
<td class="tg-c3ow" rowspan="2">written</td>
|
| 61 |
+
<td class="tg-0pky"><a href="https://corpus.korean.go.kr/" target="_blank" rel="noopener noreferrer">NIKL Newspaper corpus</a></td>
|
| 62 |
+
<td class="tg-c3ow" rowspan="2">15GB</td>
|
| 63 |
+
</tr>
|
| 64 |
+
<tr>
|
| 65 |
+
<td class="tg-0pky"><a href="https://github.com/lovit/namuwikitext" target="_blank" rel="noopener noreferrer">namuwikitext</a></td>
|
| 66 |
+
</tr>
|
| 67 |
+
</tbody>
|
| 68 |
+
</table>
|
| 69 |
+
|
| 70 |
+
<br>
|
| 71 |
+
|
| 72 |
+
## Vocabulary
|
| 73 |
+
|
| 74 |
+
We applied morpheme analysis using [huggingface_konlpy](https://github.com/lovit/huggingface_konlpy) when creating a vocabulary dictionary.
|
| 75 |
+
As a result of the experiment, it showed better performance than a vocabulary dictionary created without applying morpheme analysis.
|
| 76 |
+
<table>
|
| 77 |
+
<thead>
|
| 78 |
+
<tr>
|
| 79 |
+
<th>vocabulary size</th>
|
| 80 |
+
<th>unused token size</th>
|
| 81 |
+
<th>limit alphabet</th>
|
| 82 |
+
<th>min frequency</th>
|
| 83 |
+
</tr>
|
| 84 |
+
</thead>
|
| 85 |
+
<tbody>
|
| 86 |
+
<tr>
|
| 87 |
+
<td>40,000</td>
|
| 88 |
+
<td>500</td>
|
| 89 |
+
<td>6,000</td>
|
| 90 |
+
<td>3</td>
|
| 91 |
+
</tr>
|
| 92 |
+
</tbody>
|
| 93 |
+
</table>
|
| 94 |
+
|
| 95 |
+
<br>
|