Hugging Face's logo

lots-o
/
ko-albert-large-v1

Fill-Mask
Transformers
PyTorch
Korean
albert
Inference Endpoints
Model card Files Community
lots-o commited on
Commit
7760b1f
•
1 Parent(s): 7ee61c3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +63 -3
README.md CHANGED
@@ -1,3 +1,63 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - ko
5
+ ---
6
+
7
+ # Korean ALBERT
8
+
9
+ # Dataset
10
+ - [AI-HUB](https://www.aihub.or.kr/)
11
+ - [국립국어원 - 모두의 말뭉치](https://kli.korean.go.kr/corpus/main/requestMain.do?lang=ko)
12
+ - [Korean News Comments](https://www.kaggle.com/junbumlee/kcbert-pretraining-corpus-korean-news-comments)
13
+
14
+
15
+ # Evaluation results
16
+ - The code for finetuning can be found at [KcBERT-Finetune](https://github.com/Beomi/KcBERT-finetune).
17
+
18
+ | | Size(용량) | Average Score | **NSMC**<br/>(acc) | **Naver NER**<br/>(F1) | **PAWS**<br/>(acc) | **KorNLI**<br/>(acc) | **KorSTS**<br/>(spearman) | **Question Pair**<br/>(acc) | **KorQuaD (Dev)**<br/>(EM/F1) |
19
+ |:---------------------- |:----------:|:-------------:|:------------------:|:----------------------:|:------------------:|:--------------------:|:-------------------------:|:---------------------------:|:-----------------------------:|
20
+ | KcELECTRA-base | 475M | 84.84 | 91.71 | 86.90 | 74.80 | 81.65 | 82.65 | **95.78** | 70.60 / 90.11 |
21
+ | KcELECTRA-base-v2022 | 475M | 85.20 | **91.97** | **87.35** | 76.50 | **82.12** | **83.67** | 95.12 | 69.00 / 90.40 |
22
+ | KcBERT-Base | 417M | 79.65 | 89.62 | 84.34 | 66.95 | 74.85 | 75.57 | 93.93 | 60.25 / 84.39 |
23
+ | KcBERT-Large | 1.2G | 81.33 | 90.68 | 85.53 | 70.15 | 76.99 | 77.49 | 94.06 | 62.16 / 86.64 |
24
+ | KoBERT | 351M | 82.21 | 89.63 | 86.11 | 80.65 | 79.00 | 79.64 | 93.93 | 52.81 / 80.27 |
25
+ | XLM-Roberta-Base | 1.03G | 84.01 | 89.49 | 86.26 | 82.95 | 79.92 | 79.09 | 93.53 | 64.70 / 88.94 |
26
+ | HanBERT | 614M | 86.24 | 90.16 | 87.31 | 82.40 | 80.89 | 83.33 | 94.19 | 78.74 / 92.02 |
27
+ | KoELECTRA-Base | 423M | 84.66 | 90.21 | 86.87 | 81.90 | 80.85 | 83.21 | 94.20 | 61.10 / 89.59 |
28
+ | KoELECTRA-Base-v2 | 423M | **86.96** | 89.70 | 87.02 | **83.90** | 80.61 | 84.30 | 94.72 | **84.34 / 92.58** |
29
+ | DistilKoBERT | 108M | 76.76 | 88.41 | 84.13 | 62.55 | 70.55 | 73.21 | 92.48 | 54.12 / 77.80 |
30
+ | **ko-albert-base-v1** | **51M** | 80.46 | 86.83 | 82.26 | 69.95 | 74.17 | 74.48 | 94.06 | 76.08 / 86.82 |
31
+ | **ko-albert-large-v1** | **75M** | 82.39 | 86.91 | 83.12 | 76.10 | 76.01 | 77.46 | 94.33 | 77.64 / 87.99 |
32
+
33
+ *The size of HanBERT is the sum of the BERT model and the tokenizer DB.
34
+
35
+ *These results were obtained using the default configuration settings. Better performance may be achieved with additional hyperparameter tuning.
36
+
37
+
38
+ # How to use
39
+
40
+ ```python
41
+ from transformers import AutoTokenizer, AutoModel
42
+
43
+ # Base Model (51M)
44
+ tokenizer = AutoTokenizer.from_pretrained("lots-o/ko-albert-base-v1")
45
+ model = AutoModel.from_pretrained("lots-o/ko-albert-base-v1")
46
+
47
+ # Large Model (75M)
48
+ tokenizer = AutoTokenizer.from_pretrained("lots-o/ko-albert-large-v1")
49
+ model = AutoModel.from_pretrained("lots-o/ko-albert-large-v1")
50
+ ```
51
+
52
+ # Acknowledgement
53
+ - The GCP/TPU environment used for training the ALBERT Model was supported by the [TRC](https://sites.research.google/trc/about/) program.
54
+
55
+ # Reference
56
+ ## Paper
57
+ - [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942)
58
+
59
+ ## Github Repos
60
+ - [google-albert](https://github.com/google-research/albert)
61
+ - [albert-zh](https://github.com/brightmart/albert_zh)
62
+ - [KcBERT](https://github.com/Beomi/KcBERT)
63
+ - [KcBERT-Finetune](https://github.com/Beomi/KcBERT-finetune)