File size: 3,992 Bytes
5bf704f 5145d38 5bf704f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
# Dialog-KoELECTRA
Github : [https://github.com/skplanet/Dialog-KoELECTRA](https://github.com/skplanet/Dialog-KoELECTRA)
## Introduction
**Dialog-KoELECTRA** is a language model specialized for dialogue. It was trained with 22GB colloquial and written style Korean text data. Dialog-ELECTRA model is made based on the [ELECTRA](https://openreview.net/pdf?id=r1xMH1BtvB) model. ELECTRA is a method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU.
<br>
## Released Models
We are initially releasing small version pre-trained model.
The model was trained on Korean text. We hope to release other models, such as base/large models, in the future.
| Model | Layers | Hidden Size | Params | Max<br/>Seq Len | Learning<br/>Rate | Batch Size | Train Steps |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Dialog-KoELECTRA-Small | 12 | 256 | 14M | 128 | 1e-4 | 512 | 700K |
<br>
## Model Performance
Dialog-KoELECTRA shows strong performance in conversational downstream tasks.
| | **NSMC**<br/>(acc) | **Question Pair**<br/>(acc) | **Korean-Hate-Speech**<br/>(F1) | **Naver NER**<br/>(F1) | **KorNLI**<br/>(acc) | **KorSTS**<br/>(spearman) |
| :--------------------- | :----------------: | :--------------------: | :----------------: | :------------------: | :-----------------------: | :-------------------------: |
| DistilKoBERT | 88.60 | 92.48 | 60.72 | 84.65 | 72.00 | 72.59 |
| **Dialog-KoELECTRA-Small** | **90.01** | **94.99** | **68.26** | **85.51** | **78.54** | **78.96** |
<br>
## Train Data
<table class="tg">
<thead>
<tr>
<th class="tg-c3ow"></th>
<th class="tg-c3ow">corpus name</th>
<th class="tg-c3ow">size</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-c3ow" rowspan="4">dialog</td>
<td class="tg-0pky"><a href="https://aihub.or.kr/aidata/85" target="_blank" rel="noopener noreferrer">Aihub Korean dialog corpus</a></td>
<td class="tg-c3ow" rowspan="4">7GB</td>
</tr>
<tr>
<td class="tg-0pky"><a href="https://corpus.korean.go.kr/" target="_blank" rel="noopener noreferrer">NIKL Spoken corpus</a></td>
</tr>
<tr>
<td class="tg-0pky"><a href="https://github.com/songys/Chatbot_data" target="_blank" rel="noopener noreferrer">Korean chatbot data</a></td>
</tr>
<tr>
<td class="tg-0pky"><a href="https://github.com/Beomi/KcBERT" target="_blank" rel="noopener noreferrer">KcBERT</a></td>
</tr>
<tr>
<td class="tg-c3ow" rowspan="2">written</td>
<td class="tg-0pky"><a href="https://corpus.korean.go.kr/" target="_blank" rel="noopener noreferrer">NIKL Newspaper corpus</a></td>
<td class="tg-c3ow" rowspan="2">15GB</td>
</tr>
<tr>
<td class="tg-0pky"><a href="https://github.com/lovit/namuwikitext" target="_blank" rel="noopener noreferrer">namuwikitext</a></td>
</tr>
</tbody>
</table>
<br>
## Vocabulary
We applied morpheme analysis using [huggingface_konlpy](https://github.com/lovit/huggingface_konlpy) when creating a vocabulary dictionary.
As a result of the experiment, it showed better performance than a vocabulary dictionary created without applying morpheme analysis.
<table>
<thead>
<tr>
<th>vocabulary size</th>
<th>unused token size</th>
<th>limit alphabet</th>
<th>min frequency</th>
</tr>
</thead>
<tbody>
<tr>
<td>40,000</td>
<td>500</td>
<td>6,000</td>
<td>3</td>
</tr>
</tbody>
</table>
<br>
|