initial commit

README.md

@@ -0,0 +1,35 @@
+# Finetuning
+
+## Result
+
+### Base Model
+
+|                       | Size  | **NSMC**<br/>(acc) | **Naver NER**<br/>(F1) | **PAWS**<br/>(acc) | **KorNLI**<br/>(acc) | **KorSTS**<br/>(spearman) | **Question Pair**<br/>(acc) | **KorQuaD (Dev)**<br/>(EM/F1) | **Korean-Hate-Speech (Dev)**<br/>(F1) |
+| :-------------------- | :---: | :----------------: | :--------------------: | :----------------: | :------------------: | :-----------------------: | :-------------------------: | :---------------------------: | :-----------------------------------: |
+| KoBERT                | 351M  |       89.59        |         87.92          |       81.25        |        79.62         |           81.59           |            94.85            |         51.75 / 79.15         |                 66.21                 |
+| XLM-Roberta-Base      | 1.03G |       89.03        |         86.65          |       82.80        |        80.23         |           78.45           |            93.80            |         64.70 / 88.94         |                 64.06                 |
+| HanBERT               | 614M  |       90.06        |         87.70          |       82.95        |        80.32         |           82.73           |            94.72            |         78.74 / 92.02         |               68.32               |
+| KoELECTRA-Base-v3 | 431M  |     90.63      |       88.11        |     84.45      |      82.24       |         85.53         |          95.25          |       84.83 / 93.45       |                 67.61                 |
+| Soongsil-BERT | 370M  |     **91.2**      |       -        |     -      |      -       |         76         |          94          |        -      | **69** |
+
+### Small Model
+
+|                        | Size | **NSMC**<br/>(acc) | **Naver NER**<br/>(F1) | **PAWS**<br/>(acc) | **KorNLI**<br/>(acc) | **KorSTS**<br/>(spearman) | **Question Pair**<br/>(acc) | **KorQuaD (Dev)**<br/>(EM/F1) | **Korean-Hate-Speech (Dev)**<br/>(F1) |
+| :--------------------- | :--: | :----------------: | :--------------------: | :----------------: | :------------------: | :-----------------------: | :-------------------------: | :---------------------------: | :-----------------------------------: |
+| DistilKoBERT           | 108M |       88.60        |         84.65          |       60.50        |        72.00         |           72.59           |            92.48            |         54.40 / 77.97         |                 60.72                 |
+| KoELECTRA-Small-v3 | 54M  |     89.36      |       85.40        |     77.45      |      78.60       |         80.79         |          94.85          |       82.11 / 91.13       |               63.07               |
+| Soongsil-BERT | 213M  |     **90.7**      |       84        |     69.1      |      76       |         -         |          92          |        -      | **66** |
+
+## Reference
+- [Transformers Examples](https://github.com/huggingface/transformers/blob/master/examples/README.md)
+- [NSMC](https://github.com/e9t/nsmc)
+- [Naver NER Dataset](https://github.com/naver/nlp-challenge)
+- [PAWS](https://github.com/google-research-datasets/paws)
+- [KorNLI/KorSTS](https://github.com/kakaobrain/KorNLUDatasets)
+- [Question Pair](https://github.com/songys/Question_pair)
+- [KorQuad](https://korquad.github.io/category/1.0_KOR.html)
+- [Korean Hate Speech](https://github.com/kocohub/korean-hate-speech)
+- [KoELECTRA](https://github.com/monologg/KoELECTRA)
+- [KoBERT](https://github.com/SKTBrain/KoBERT)
+- [HanBERT](https://github.com/tbai2019/HanBert-54k-N)
+- [HanBert Transformers](https://github.com/monologg/HanBert-Transformers)

config.json

@@ -0,0 +1,33 @@
+{
+  "architectures": [
+    "RobertaForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "none",
+    "1": "offensive",
+    "2": "hate"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "hate": 2,
+    "none": 0,
+    "offensive": 1
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "total_flos": 8.217624121867606e+19,
+  "type_vocab_size": 1,
+  "vocab_size": 16000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bf1435f5b6f9727c9b7fbf89ba4ef5cf3e8cf68b4365d0868187f2f32ef69f7
+size 393420233

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": false}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"max_len": 512, "special_tokens_map_file": null, "full_tokenizer_file": null}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17405666aa60e70369a597d3349370e9143c329ab67f7e8f9ba33f2abb679861
+size 1519