asahi417 commited on
Commit
2fc4b33
1 Parent(s): 1ded3c1

model update

Browse files
.gitattributes CHANGED
@@ -32,3 +32,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
32
  *.zip filter=lfs diff=lfs merge=lfs -text
33
  *.zst filter=lfs diff=lfs merge=lfs -text
34
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
32
  *.zip filter=lfs diff=lfs merge=lfs -text
33
  *.zst filter=lfs diff=lfs merge=lfs -text
34
  *tfevents* filter=lfs diff=lfs merge=lfs -text
35
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+ license: cc-by-4.0
4
+ metrics:
5
+ - bleu4
6
+ - meteor
7
+ - rouge-l
8
+ - bertscore
9
+ - moverscore
10
+ language: ko
11
+ datasets:
12
+ - lmqg/qg_koquad
13
+ pipeline_tag: text2text-generation
14
+ tags:
15
+ - question generation
16
+ widget:
17
+ - text: "1990년 영화 《 <hl> 남부군 <hl> 》에서 단역으로 영화배우 첫 데뷔에 이어 같은 해 KBS 드라마 《지구인》에서 단역으로 출연하였고 이듬해 MBC 《여명의 눈동자》를 통해 단역으로 출연하였다."
18
+ example_title: "Question Generation Example 1"
19
+ - text: "백신이 없기때문에 예방책은 <hl> 살충제 <hl> 를 사용하면서 서식 장소(찻찬 받침, 배수로, 고인 물의 열린 저장소, 버려진 타이어 등)의 수를 줄임으로써 매개체를 통제할 수 있다."
20
+ example_title: "Question Generation Example 2"
21
+ - text: "<hl> 원테이크 촬영 <hl> 이기 때문에 한 사람이 실수를 하면 처음부터 다시 찍어야 하는 상황이 발생한다."
22
+ example_title: "Question Generation Example 3"
23
+ model-index:
24
+ - name: vocabtrimmer/mt5-base-trimmed-ko-15000-koquad-qg
25
+ results:
26
+ - task:
27
+ name: Text2text Generation
28
+ type: text2text-generation
29
+ dataset:
30
+ name: lmqg/qg_koquad
31
+ type: default
32
+ args: default
33
+ metrics:
34
+ - name: BLEU4 (Question Generation)
35
+ type: bleu4_question_generation
36
+ value: 11.7
37
+ - name: ROUGE-L (Question Generation)
38
+ type: rouge_l_question_generation
39
+ value: 27.43
40
+ - name: METEOR (Question Generation)
41
+ type: meteor_question_generation
42
+ value: 28.76
43
+ - name: BERTScore (Question Generation)
44
+ type: bertscore_question_generation
45
+ value: 83.92
46
+ - name: MoverScore (Question Generation)
47
+ type: moverscore_question_generation
48
+ value: 82.94
49
+ ---
50
+
51
+ # Model Card of `vocabtrimmer/mt5-base-trimmed-ko-15000-koquad-qg`
52
+ This model is fine-tuned version of [vocabtrimmer/mt5-base-trimmed-ko-15000](https://huggingface.co/vocabtrimmer/mt5-base-trimmed-ko-15000) for question generation task on the [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) (dataset_name: default) via [`lmqg`](https://github.com/asahi417/lm-question-generation).
53
+
54
+
55
+ ### Overview
56
+ - **Language model:** [vocabtrimmer/mt5-base-trimmed-ko-15000](https://huggingface.co/vocabtrimmer/mt5-base-trimmed-ko-15000)
57
+ - **Language:** ko
58
+ - **Training data:** [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) (default)
59
+ - **Online Demo:** [https://autoqg.net/](https://autoqg.net/)
60
+ - **Repository:** [https://github.com/asahi417/lm-question-generation](https://github.com/asahi417/lm-question-generation)
61
+ - **Paper:** [https://arxiv.org/abs/2210.03992](https://arxiv.org/abs/2210.03992)
62
+
63
+ ### Usage
64
+ - With [`lmqg`](https://github.com/asahi417/lm-question-generation#lmqg-language-model-for-question-generation-)
65
+ ```python
66
+ from lmqg import TransformersQG
67
+
68
+ # initialize model
69
+ model = TransformersQG(language="ko", model="vocabtrimmer/mt5-base-trimmed-ko-15000-koquad-qg")
70
+
71
+ # model prediction
72
+ questions = model.generate_q(list_context="1990년 영화 《 남부군 》에서 단역으로 영화배우 첫 데뷔에 이어 같은 해 KBS 드라마 《지구인》에서 단역으로 출연하였고 이듬해 MBC 《여명의 눈동자》를 통해 단역으로 출연하였다.", list_answer="남부군")
73
+
74
+ ```
75
+
76
+ - With `transformers`
77
+ ```python
78
+ from transformers import pipeline
79
+
80
+ pipe = pipeline("text2text-generation", "vocabtrimmer/mt5-base-trimmed-ko-15000-koquad-qg")
81
+ output = pipe("1990년 영화 《 <hl> 남부군 <hl> 》에서 단역으로 영화배우 첫 데뷔에 이어 같은 해 KBS 드라마 《지구인》에서 단역으로 출연하였고 이듬해 MBC 《여명의 눈동자》를 통해 단역으로 출연하였다.")
82
+
83
+ ```
84
+
85
+ ## Evaluation
86
+
87
+
88
+ - ***Metric (Question Generation)***: [raw metric file](https://huggingface.co/vocabtrimmer/mt5-base-trimmed-ko-15000-koquad-qg/raw/main/eval/metric.first.sentence.paragraph_answer.question.lmqg_qg_koquad.default.json)
89
+
90
+ | | Score | Type | Dataset |
91
+ |:-----------|--------:|:--------|:-----------------------------------------------------------------|
92
+ | BERTScore | 83.92 | default | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) |
93
+ | Bleu_1 | 27.39 | default | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) |
94
+ | Bleu_2 | 20.25 | default | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) |
95
+ | Bleu_3 | 15.29 | default | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) |
96
+ | Bleu_4 | 11.7 | default | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) |
97
+ | METEOR | 28.76 | default | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) |
98
+ | MoverScore | 82.94 | default | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) |
99
+ | ROUGE_L | 27.43 | default | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) |
100
+
101
+
102
+
103
+ ## Training hyperparameters
104
+
105
+ The following hyperparameters were used during fine-tuning:
106
+ - dataset_path: lmqg/qg_koquad
107
+ - dataset_name: default
108
+ - input_types: paragraph_answer
109
+ - output_types: question
110
+ - prefix_types: None
111
+ - model: vocabtrimmer/mt5-base-trimmed-ko-15000
112
+ - max_length: 512
113
+ - max_length_output: 32
114
+ - epoch: 15
115
+ - batch: 16
116
+ - lr: 0.0005
117
+ - fp16: False
118
+ - random_seed: 1
119
+ - gradient_accumulation_steps: 4
120
+ - label_smoothing: 0.15
121
+
122
+ The full configuration can be found at [fine-tuning config file](https://huggingface.co/vocabtrimmer/mt5-base-trimmed-ko-15000-koquad-qg/raw/main/trainer_config.json).
123
+
124
+ ## Citation
125
+ ```
126
+ @inproceedings{ushio-etal-2022-generative,
127
+ title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
128
+ author = "Ushio, Asahi and
129
+ Alva-Manchego, Fernando and
130
+ Camacho-Collados, Jose",
131
+ booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
132
+ month = dec,
133
+ year = "2022",
134
+ address = "Abu Dhabi, U.A.E.",
135
+ publisher = "Association for Computational Linguistics",
136
+ }
137
+
138
+ ```
added_tokens.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "<hl>": 15001
3
+ }
config.json ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "lmqg_output/trimmed_qg/mt5-base-trimmed-ko-15000-koquad-qg/model_dpyopu/epoch_5",
3
+ "add_prefix": false,
4
+ "architectures": [
5
+ "MT5ForConditionalGeneration"
6
+ ],
7
+ "d_ff": 2048,
8
+ "d_kv": 64,
9
+ "d_model": 768,
10
+ "decoder_start_token_id": 0,
11
+ "dense_act_fn": "gelu_new",
12
+ "dropout_rate": 0.1,
13
+ "eos_token_id": 1,
14
+ "feed_forward_proj": "gated-gelu",
15
+ "initializer_factor": 1.0,
16
+ "is_encoder_decoder": true,
17
+ "is_gated_act": true,
18
+ "layer_norm_epsilon": 1e-06,
19
+ "model_type": "mt5",
20
+ "num_decoder_layers": 12,
21
+ "num_heads": 12,
22
+ "num_layers": 12,
23
+ "output_past": true,
24
+ "pad_token_id": 0,
25
+ "relative_attention_max_distance": 128,
26
+ "relative_attention_num_buckets": 32,
27
+ "tie_word_embeddings": false,
28
+ "tokenizer_class": "T5Tokenizer",
29
+ "torch_dtype": "float32",
30
+ "transformers_version": "4.26.1",
31
+ "use_cache": true,
32
+ "vocab_size": 15002,
33
+ "vocabtrimmer": {
34
+ "mining_config": {
35
+ "dataset": "mc4",
36
+ "dataset_column": "text",
37
+ "dataset_name": "ko",
38
+ "dataset_split": "validation",
39
+ "language": "ko",
40
+ "min_frequency": 2,
41
+ "target_vocab_size": 15000
42
+ },
43
+ "stats": {
44
+ "compression_rate_embedding": 5.9977130245649946,
45
+ "compression_rate_full": 37.992839576176756,
46
+ "parameter_size_embedding/raw": 384172032,
47
+ "parameter_size_embedding/trimmed": 23041536,
48
+ "parameter_size_full/raw": 582401280,
49
+ "parameter_size_full/trimmed": 221270784,
50
+ "vocab_size/raw": 250112,
51
+ "vocab_size/trimmed": 15001
52
+ }
53
+ }
54
+ }
eval/metric.first.answer.paragraph_answer.question.lmqg_qg_koquad.default.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"validation": {"Bleu_1": 0.2550327307052953, "Bleu_2": 0.18565919602820272, "Bleu_3": 0.13849787805956632, "Bleu_4": 0.10455582517645232}, "test": {"Bleu_1": 0.27074135776759123, "Bleu_2": 0.19984438513176458, "Bleu_3": 0.1508790452307565, "Bleu_4": 0.11543086461884584}}
eval/metric.first.sentence.paragraph_answer.question.lmqg_qg_koquad.default.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"validation": {"Bleu_1": 0.2874901609532262, "Bleu_2": 0.21295401906754355, "Bleu_3": 0.16086704096760465, "Bleu_4": 0.12262995630720323, "METEOR": 0.2906756591561873, "ROUGE_L": 0.27981872333332347, "BERTScore": 0.8312479596810421, "MoverScore": 0.8296273142630661}, "test": {"Bleu_1": 0.2739280981490131, "Bleu_2": 0.20249457516259292, "Bleu_3": 0.15293624325491237, "Bleu_4": 0.11698815032804398, "METEOR": 0.28759461092604915, "ROUGE_L": 0.27430471861095135, "BERTScore": 0.8392473354346395, "MoverScore": 0.8294259768954083}}
eval/samples.test.hyp.paragraph_answer.question.lmqg_qg_koquad.default.txt ADDED
The diff for this file is too large to render. See raw diff
 
eval/samples.validation.hyp.paragraph_answer.question.lmqg_qg_koquad.default.txt ADDED
The diff for this file is too large to render. See raw diff
 
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "decoder_start_token_id": 0,
4
+ "eos_token_id": 1,
5
+ "pad_token_id": 0,
6
+ "transformers_version": "4.26.1"
7
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7a23b3d9aa8ced5252ffa74542d6fdbeba7237177212274e19e0139d0f5711be
3
+ size 885186613
special_tokens_map.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<hl>"
4
+ ],
5
+ "eos_token": "</s>",
6
+ "pad_token": "<pad>",
7
+ "unk_token": "<unk>"
8
+ }
spiece.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef78f86560d809067d12bac6c09f19a462cb3af3f54d2b8acbba26e1433125d6
3
+ size 4309802
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2d19ac9dd9ec96dd229b5d13eef16e28f3f5c930987123e2f770dd6f6140aa4d
3
+ size 1244356
tokenizer_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": null,
3
+ "eos_token": "</s>",
4
+ "extra_ids": 0,
5
+ "model_max_length": 1000000000000000019884624838656,
6
+ "name_or_path": "lmqg_output/trimmed_qg/mt5-base-trimmed-ko-15000-koquad-qg/model_dpyopu/epoch_5",
7
+ "pad_token": "<pad>",
8
+ "sp_model_kwargs": {},
9
+ "special_tokens_map_file": "/home/patrick/.cache/torch/transformers/685ac0ca8568ec593a48b61b0a3c272beee9bc194a3c7241d15dcadb5f875e53.f76030f3ec1b96a8199b2593390c610e76ca8028ef3d24680000619ffb646276",
10
+ "tokenizer_class": "T5Tokenizer",
11
+ "unk_token": "<unk>"
12
+ }
trainer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"dataset_path": "lmqg/qg_koquad", "dataset_name": "default", "input_types": "paragraph_answer", "output_types": "question", "prefix_types": null, "model": "vocabtrimmer/mt5-base-trimmed-ko-15000", "max_length": 512, "max_length_output": 32, "epoch": 15, "batch": 16, "lr": 0.0005, "fp16": false, "random_seed": 1, "gradient_accumulation_steps": 4, "label_smoothing": 0.15}