yhavinga commited on
Commit
5f09441
1 Parent(s): 784fb51
README.md ADDED
@@ -0,0 +1,197 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+ language:
4
+ - nl
5
+ - en
6
+ - multilingual
7
+ license: apache-2.0
8
+ tags:
9
+ - dutch
10
+ - english
11
+ - t5
12
+ - t5x
13
+ - ul2
14
+ - seq2seq
15
+ - translation
16
+ datasets:
17
+ - yhavinga/mc4_nl_cleaned
18
+ - yhavinga/nedd_wiki_news
19
+ pipeline_tag: translation
20
+ widget:
21
+ - text: >-
22
+ Redistricting and West Virginia’s shrinking population forced the state’s
23
+ Republican Legislature to pit Mr. McKinley, a six-term Republican with a
24
+ pragmatic bent, against Mr. Mooney, who has served four terms marked more
25
+ by conservative rhetoric than legislative achievements.
26
+ - text: >-
27
+ It is a painful and tragic spectacle that rises before me: I have drawn
28
+ back the curtain from the rottenness of man. This word, in my mouth, is at
29
+ least free from one suspicion: that it involves a moral accusation against
30
+ humanity.
31
+ - text: >-
32
+ Young Wehling was hunched in his chair, his head in his hand. He was so
33
+ rumpled, so still and colorless as to be virtually invisible. His
34
+ camouflage was perfect, since the waiting room had a disorderly and
35
+ demoralized air, too. Chairs and ashtrays had been moved away from the
36
+ walls. The floor was paved with spattered dropcloths.
37
+ ---
38
+
39
+ # ul2-large-en-nl for English to Dutch translation
40
+
41
+ Fine-tuned T5 model on English to Dutch translation that was pretrained on Dutch using a UL2 (Mixture-of-Denoisers) objective.
42
+ The T5 model was introduced in
43
+ [this paper](https://arxiv.org/abs/1910.10683)
44
+ and first released at [this page](https://github.com/google-research/text-to-text-transfer-transformer).
45
+ The UL2 objective was introduced in
46
+ [this paper](https://arxiv.org/abs/2205.05131)
47
+ and first released at [this page](https://github.com/google-research/google-research/tree/master/ul2).
48
+
49
+
50
+
51
+ ## Model description
52
+
53
+ T5 is an encoder-decoder model and treats all NLP problems in a text-to-text format.
54
+
55
+ `ul2-large-en-nl` T5 is a transformers model fine-tuned on parallel sentence and paragraph pairs
56
+ sampled from books.
57
+
58
+ This model used the [T5 v1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511) improvements compared to the original T5 model during the pretraining:
59
+ - GEGLU activation in the feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202)
60
+ - Dropout was turned off during pre-training. Dropout should be re-enabled during fine-tuning
61
+ - Pre-trained on self-supervised objective only without mixing in the downstream tasks
62
+ - No parameter sharing between embedding and classifier layer
63
+
64
+
65
+
66
+ ### UL2 pretraining objective
67
+
68
+ This model was pretrained with the UL2's Mixture-of-Denoisers (MoD) objective, that combines diverse pre-training
69
+ paradigms together. UL2 frames different objective functions for training language models as denoising tasks, where
70
+ the model has to recover missing sub-sequences of a given input. During pre-training it uses a novel mixture-of-denoisers
71
+ that samples from a varied set of such objectives, each with different configurations. UL2 is trained using a mixture of
72
+ three denoising tasks:
73
+
74
+ 1. R-denoising (or regular span corruption), which emulates the standard T5 span corruption objective;
75
+ 2. X-denoising (or extreme span corruption); and
76
+ 3. S-denoising (or sequential PrefixLM).
77
+
78
+ During pre-training, we sample from the available denoising tasks based on user-specified ratios.
79
+ UL2 introduces a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training
80
+ denoising task. During the pre-training, a paradigm token is inserted to the input
81
+ (`[NLU]` for R-denoising, `[NLG]` for X-denoising, or `[S2S]` for S-denoising) indicating the denoising task at hand.
82
+ Then, during fine-tuning the same input token should be inserted to get the best performance for different downstream
83
+ fine-tuning tasks.
84
+
85
+ ## Intended uses & limitations
86
+
87
+ This model was fine-tuned on parallel sentence and paragraph pairs and can be used
88
+ for machine translation.
89
+
90
+ ### How to use
91
+
92
+ Here is how to use this model in PyTorch:
93
+
94
+ ```python
95
+ model_name = "yhavinga/ul2-large-en-nl"
96
+ from transformers import AutoTokenizer
97
+ from transformers import AutoModelForSeq2SeqLM
98
+ from transformers import pipeline
99
+ import torch
100
+ device_num = 0 if torch.cuda.is_available() else -1
101
+ device = "cpu" if device_num < 0 else f"cuda:{device_num}"
102
+
103
+ tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
104
+ model = AutoModelForSeq2SeqLM.from_pretrained(model_name, use_auth_token=True).to(
105
+ device
106
+ )
107
+ params = {"max_length": 370, "num_beams": 4, "early_stopping": True}
108
+ translator = pipeline("translation", tokenizer=tokenizer, model=model, device=device_num)
109
+ print(translator("Young Wehling was hunched in his chair, his head in his hand. He was so rumpled, so still and colorless as to be virtually invisible.",
110
+ **params)[0]['translation_text'])
111
+ ```
112
+
113
+
114
+ ### Limitations and bias
115
+
116
+ The training data used for this model contains a lot of unfiltered content from the internet, which is far from neutral.
117
+ Therefore, the model can have biased predictions. This bias will also affect all fine-tuned versions of this model.
118
+
119
+ ## Training data
120
+
121
+ The `ul2-large-en-nl` T5 model was pre-trained simultaneously on a combination of several datasets,
122
+ including the `full` config of the "mc4_nl_cleaned" dataset, which is a cleaned version of Common Crawl's web
123
+ crawl corpus, Dutch books, the Dutch subset of Wikipedia (2022-03-20), and a subset of "mc4_nl_cleaned"
124
+ containing only texts from Dutch and Belgian newspapers. This last dataset is oversampled to bias the model
125
+ towards descriptions of events in the Netherlands and Belgium.
126
+
127
+ After pre-training, the model was
128
+ fine-tuned on a translation dataset containing 13 million sentence and paragraph pairs
129
+ sampled from books.
130
+
131
+
132
+
133
+ ## Training procedure
134
+
135
+ ### Preprocessing
136
+
137
+ The ul2-large-en-nl T5 model uses a SentencePiece unigram tokenizer with a vocabulary of 32,000 tokens.
138
+ The tokenizer includes the special tokens `<pad>`, `</s>`, `<unk>`, known from the original T5 paper,
139
+ `[NLU]`, `[NLG]` and `[S2S]` for the MoD pre-training, and `<n>` for newline.
140
+ During pre-training with the UL2 objective, input and output sequences consist of 512 consecutive tokens.
141
+ The tokenizer does not lowercase texts and is therefore case-sensitive; it distinguises
142
+ between `dutch` and `Dutch`.
143
+ Additionally, 100+28 extra tokens were added for pre-training tasks, resulting in a total of 32,128 tokens.
144
+
145
+ ### Fine-tuning
146
+
147
+ This model was fine-tuned on a dataset containing 13M sentence and paragraph translation pairs sampled from books.
148
+
149
+ * Pre-trained model used as starting point: yhavinga/ul2-large-dutch
150
+ * Amount of fine-tune training steps: 77600
151
+ * Batch size: 512 (gradient accumulation steps: 16)
152
+ * Sequence length: 370 tokens
153
+ * Model dtype: bfloat16
154
+ * z_loss: 0.0001
155
+ * Optimizer: adamw_hf beta1: 0.9 beta2: 0.9969 eps: 1e-08
156
+ * Dropout rate: 0.01
157
+ * Learning rate: 0.0009 with linear decay to 0 and warmup for 500 steps
158
+ * Label smoothing factor: 0.11
159
+ * Bleu score: 45.1
160
+
161
+ ### Model list
162
+
163
+ Models in this series:
164
+
165
+
166
+ | | ul2-base-en-nl | ul2-base-nl36-en-nl | ul2-large-en-nl |
167
+ |:---------------------|:-----------------|:----------------------|:------------------|
168
+ | model_type | t5 | t5 | t5 |
169
+ | _pipeline_tag | translation | translation | translation |
170
+ | d_model | 768 | 768 | 1024 |
171
+ | d_ff | 2048 | 3072 | 2816 |
172
+ | num_heads | 12 | 12 | 16 |
173
+ | d_kv | 64 | 64 | 64 |
174
+ | num_layers | 12 | 36 | 24 |
175
+ | num_decoder_layers | 12 | 36 | 24 |
176
+ | feed_forward_proj | gated-silu | gated-silu | gated-silu |
177
+ | dense_act_fn | silu | silu | silu |
178
+ | vocab_size | 32128 | 32128 | 32128 |
179
+ | tie_word_embeddings | 0 | 0 | 0 |
180
+ | torch_dtype | float32 | float32 | float32 |
181
+ | _gin_batch_size | 128 | 64 | 64 |
182
+ | _gin_z_loss | 0.0001 | 0.0001 | 0.0001 |
183
+ | _gin_t5_config_dtype | 'bfloat16' | 'bfloat16' | 'bfloat16' |
184
+
185
+ ## Evaluation results
186
+
187
+ See the evaluation section in the interactive [Pre-training Dutch T5 Models](https://huggingface.co/spaces/yhavinga/pre-training-dutch-t5-models) blog.
188
+
189
+ ## Acknowledgements
190
+
191
+ This project would not have been possible without compute generously provided by Google through the
192
+ [TPU Research Cloud](https://sites.research.google/trc/).
193
+ Thanks to the [Finnish-NLP](https://huggingface.co/Finnish-NLP) authors for releasing their code for the UL2 objective and associated task definitions.
194
+ Thanks to [Stephenn Fernandes](https://huggingface.co/StephennFernandes) for helping me get started with the t5x framework.
195
+
196
+ Created by [Yeb Havinga](https://www.linkedin.com/in/yeb-havinga-86530825/)
197
+
added_tokens.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"[new_id_17]": 32117, "[new_id_20]": 32120, "[new_id_13]": 32113, "[new_id_2]": 32102, "[new_id_16]": 32116, "[new_id_7]": 32107, "[new_id_5]": 32105, "[new_id_1]": 32101, "[new_id_15]": 32115, "[new_id_12]": 32112, "[new_id_0]": 32100, "[new_id_11]": 32111, "[new_id_25]": 32125, "[new_id_24]": 32124, "[new_id_10]": 32110, "[new_id_27]": 32127, "[new_id_23]": 32123, "[new_id_14]": 32114, "[new_id_22]": 32122, "[new_id_21]": 32121, "[new_id_19]": 32119, "[new_id_3]": 32103, "[new_id_4]": 32104, "[new_id_18]": 32118, "[new_id_9]": 32109, "[new_id_8]": 32108, "[new_id_26]": 32126, "[new_id_6]": 32106}
config.gin ADDED
@@ -0,0 +1,150 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __gin__ import dynamic_registration
2
+ import __main__ as train_script
3
+ import seqio
4
+ import t5.data.mixtures
5
+ from t5x import adafactor
6
+ from t5x.examples.t5 import network
7
+ from t5x import gin_utils
8
+ from t5x import models
9
+ from t5x import partitioning
10
+ from t5x import trainer
11
+ from t5x import utils
12
+ import tasks.nedd_tasks
13
+ import tasks.ul2_tasks as tasks2
14
+
15
+ # Macros:
16
+ # ==============================================================================
17
+ BATCH_SIZE = 64
18
+ DROPOUT_RATE = 0.0
19
+ LABEL_SMOOTHING = 0.0
20
+ LOSS_NORMALIZING_FACTOR = None
21
+ MIXTURE_OR_TASK_MODULE = None
22
+ MIXTURE_OR_TASK_NAME = 'ul2_mc4_nedd_wiki_news_mix_1'
23
+ MODEL = @models.EncoderDecoderModel()
24
+ MODEL_DIR = 'ul2_large_mc4_nedd_wiki_news_nl'
25
+ OPTIMIZER = @adafactor.Adafactor()
26
+ RANDOM_SEED = None
27
+ SHUFFLE_TRAIN_EXAMPLES = True
28
+ TASK_FEATURE_LENGTHS = {'inputs': 512, 'targets': 512}
29
+ TRAIN_STEPS = 1000000
30
+ USE_CACHED_TASKS = False
31
+ USE_HARDWARE_RNG = False
32
+ VOCABULARY = @seqio.SentencePieceVocabulary()
33
+ Z_LOSS = 0.0001
34
+
35
+ # Parameters for adafactor.Adafactor:
36
+ # ==============================================================================
37
+ adafactor.Adafactor.decay_rate = 0.8
38
+ adafactor.Adafactor.logical_factor_rules = \
39
+ @adafactor.standard_logical_factor_rules()
40
+ adafactor.Adafactor.step_offset = 0
41
+
42
+ # Parameters for utils.CheckpointConfig:
43
+ # ==============================================================================
44
+ utils.CheckpointConfig.restore = @utils.RestoreCheckpointConfig()
45
+ utils.CheckpointConfig.save = @utils.SaveCheckpointConfig()
46
+
47
+ # Parameters for utils.create_learning_rate_scheduler:
48
+ # ==============================================================================
49
+ utils.create_learning_rate_scheduler.base_learning_rate = 1.0
50
+ utils.create_learning_rate_scheduler.factors = 'constant * rsqrt_decay'
51
+ utils.create_learning_rate_scheduler.warmup_steps = 10000
52
+
53
+ # Parameters for train/utils.DatasetConfig:
54
+ # ==============================================================================
55
+ train/utils.DatasetConfig.batch_size = %BATCH_SIZE
56
+ train/utils.DatasetConfig.mixture_or_task_name = %MIXTURE_OR_TASK_NAME
57
+ train/utils.DatasetConfig.module = %MIXTURE_OR_TASK_MODULE
58
+ train/utils.DatasetConfig.pack = True
59
+ train/utils.DatasetConfig.seed = None
60
+ train/utils.DatasetConfig.shuffle = %SHUFFLE_TRAIN_EXAMPLES
61
+ train/utils.DatasetConfig.split = 'train'
62
+ train/utils.DatasetConfig.task_feature_lengths = %TASK_FEATURE_LENGTHS
63
+ train/utils.DatasetConfig.use_cached = %USE_CACHED_TASKS
64
+
65
+ # Parameters for train_eval/utils.DatasetConfig:
66
+ # ==============================================================================
67
+ train_eval/utils.DatasetConfig.batch_size = %BATCH_SIZE
68
+ train_eval/utils.DatasetConfig.mixture_or_task_name = %MIXTURE_OR_TASK_NAME
69
+ train_eval/utils.DatasetConfig.module = %MIXTURE_OR_TASK_MODULE
70
+ train_eval/utils.DatasetConfig.pack = True
71
+ train_eval/utils.DatasetConfig.seed = 42
72
+ train_eval/utils.DatasetConfig.shuffle = False
73
+ train_eval/utils.DatasetConfig.split = 'validation'
74
+ train_eval/utils.DatasetConfig.task_feature_lengths = %TASK_FEATURE_LENGTHS
75
+ train_eval/utils.DatasetConfig.use_cached = %USE_CACHED_TASKS
76
+
77
+ # Parameters for models.EncoderDecoderModel:
78
+ # ==============================================================================
79
+ models.EncoderDecoderModel.input_vocabulary = %VOCABULARY
80
+ models.EncoderDecoderModel.label_smoothing = %LABEL_SMOOTHING
81
+ models.EncoderDecoderModel.loss_normalizing_factor = %LOSS_NORMALIZING_FACTOR
82
+ models.EncoderDecoderModel.module = @network.Transformer()
83
+ models.EncoderDecoderModel.optimizer_def = %OPTIMIZER
84
+ models.EncoderDecoderModel.output_vocabulary = %VOCABULARY
85
+ models.EncoderDecoderModel.z_loss = %Z_LOSS
86
+
87
+ # Parameters for partitioning.PjitPartitioner:
88
+ # ==============================================================================
89
+ partitioning.PjitPartitioner.logical_axis_rules = \
90
+ @partitioning.standard_logical_axis_rules()
91
+ partitioning.PjitPartitioner.model_parallel_submesh = None
92
+ partitioning.PjitPartitioner.num_partitions = 1
93
+
94
+ # Parameters for utils.RestoreCheckpointConfig:
95
+ # ==============================================================================
96
+ utils.RestoreCheckpointConfig.path = []
97
+
98
+ # Parameters for utils.SaveCheckpointConfig:
99
+ # ==============================================================================
100
+ utils.SaveCheckpointConfig.dtype = 'float32'
101
+ utils.SaveCheckpointConfig.keep = 4
102
+ utils.SaveCheckpointConfig.period = 50000
103
+ utils.SaveCheckpointConfig.save_dataset = False
104
+ utils.SaveCheckpointConfig.use_gda = False
105
+
106
+ # Parameters for seqio.SentencePieceVocabulary:
107
+ # ==============================================================================
108
+ seqio.SentencePieceVocabulary.sentencepiece_model_file = \
109
+ 'gs://t5-dutch-english/vocabs/nedd.32000.128extra/spiece.model'
110
+
111
+ # Parameters for network.T5Config:
112
+ # ==============================================================================
113
+ network.T5Config.dropout_rate = %DROPOUT_RATE
114
+ network.T5Config.dtype = 'bfloat16'
115
+ network.T5Config.emb_dim = 1024
116
+ network.T5Config.head_dim = 64
117
+ network.T5Config.logits_via_embedding = False
118
+ network.T5Config.mlp_activations = ('gelu', 'linear')
119
+ network.T5Config.mlp_dim = 2816
120
+ network.T5Config.num_decoder_layers = 24
121
+ network.T5Config.num_encoder_layers = 24
122
+ network.T5Config.num_heads = 16
123
+ network.T5Config.vocab_size = 32128
124
+
125
+ # Parameters for train_script.train:
126
+ # ==============================================================================
127
+ train_script.train.checkpoint_cfg = @utils.CheckpointConfig()
128
+ train_script.train.eval_period = 2000
129
+ train_script.train.eval_steps = 20
130
+ train_script.train.infer_eval_dataset_cfg = None
131
+ train_script.train.model = %MODEL
132
+ train_script.train.model_dir = %MODEL_DIR
133
+ train_script.train.partitioner = @partitioning.PjitPartitioner()
134
+ train_script.train.random_seed = %RANDOM_SEED
135
+ train_script.train.stats_period = 100
136
+ train_script.train.summarize_config_fn = @gin_utils.summarize_gin_config
137
+ train_script.train.total_steps = %TRAIN_STEPS
138
+ train_script.train.train_dataset_cfg = @train/utils.DatasetConfig()
139
+ train_script.train.train_eval_dataset_cfg = @train_eval/utils.DatasetConfig()
140
+ train_script.train.trainer_cls = @trainer.Trainer
141
+ train_script.train.use_hardware_rng = %USE_HARDWARE_RNG
142
+
143
+ # Parameters for trainer.Trainer:
144
+ # ==============================================================================
145
+ trainer.Trainer.learning_rate_fn = @utils.create_learning_rate_scheduler()
146
+ trainer.Trainer.num_microbatches = None
147
+
148
+ # Parameters for network.Transformer:
149
+ # ==============================================================================
150
+ network.Transformer.config = @network.T5Config()
config.json ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "yhavinga/ul2-large-en-nl",
3
+ "architectures": [
4
+ "T5ForConditionalGeneration"
5
+ ],
6
+ "d_ff": 2816,
7
+ "d_kv": 64,
8
+ "d_model": 1024,
9
+ "decoder_start_token_id": 0,
10
+ "dense_act_fn": "silu",
11
+ "dropout_rate": 0.01,
12
+ "eos_token_id": 1,
13
+ "early_stopping": true,
14
+ "feed_forward_proj": "gated-silu",
15
+ "initializer_factor": 1.0,
16
+ "is_encoder_decoder": true,
17
+ "is_gated_act": true,
18
+ "layer_norm_epsilon": 1e-06,
19
+ "max_length": 370,
20
+ "model_type": "t5",
21
+ "n_positions": 512,
22
+ "num_beams": 4,
23
+ "num_decoder_layers": 24,
24
+ "num_heads": 16,
25
+ "num_layers": 24,
26
+ "output_past": true,
27
+ "pad_token_id": 0,
28
+ "relative_attention_max_distance": 128,
29
+ "relative_attention_num_buckets": 32,
30
+ "tie_word_embeddings": false,
31
+ "torch_dtype": "float32",
32
+ "transformers_version": "4.23.1",
33
+ "use_cache": true,
34
+ "vocab_size": 32128
35
+ }
events.out.tfevents.1672813339.t1v-n-c82e3785-w-0.2029925.0.v2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac1d98fc1bccf791ede706b7a1dcfd34f998c772b2f8deba7c73ec9de5c2271c
3
+ size 3472065
flax_model.msgpack ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0943ba2f958f2d04dec07e54975e3aae3c00a933dc752cef4d9787ab4c264588
3
+ size 1632372682
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67ccb5db4ce777787f9147bb8d5d941ef6c3d4cc0fdbacf02209ca52622d438a
3
+ size 3132781861
special_tokens_map.json ADDED
@@ -0,0 +1,107 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<extra_id_0>",
4
+ "<extra_id_1>",
5
+ "<extra_id_2>",
6
+ "<extra_id_3>",
7
+ "<extra_id_4>",
8
+ "<extra_id_5>",
9
+ "<extra_id_6>",
10
+ "<extra_id_7>",
11
+ "<extra_id_8>",
12
+ "<extra_id_9>",
13
+ "<extra_id_10>",
14
+ "<extra_id_11>",
15
+ "<extra_id_12>",
16
+ "<extra_id_13>",
17
+ "<extra_id_14>",
18
+ "<extra_id_15>",
19
+ "<extra_id_16>",
20
+ "<extra_id_17>",
21
+ "<extra_id_18>",
22
+ "<extra_id_19>",
23
+ "<extra_id_20>",
24
+ "<extra_id_21>",
25
+ "<extra_id_22>",
26
+ "<extra_id_23>",
27
+ "<extra_id_24>",
28
+ "<extra_id_25>",
29
+ "<extra_id_26>",
30
+ "<extra_id_27>",
31
+ "<extra_id_28>",
32
+ "<extra_id_29>",
33
+ "<extra_id_30>",
34
+ "<extra_id_31>",
35
+ "<extra_id_32>",
36
+ "<extra_id_33>",
37
+ "<extra_id_34>",
38
+ "<extra_id_35>",
39
+ "<extra_id_36>",
40
+ "<extra_id_37>",
41
+ "<extra_id_38>",
42
+ "<extra_id_39>",
43
+ "<extra_id_40>",
44
+ "<extra_id_41>",
45
+ "<extra_id_42>",
46
+ "<extra_id_43>",
47
+ "<extra_id_44>",
48
+ "<extra_id_45>",
49
+ "<extra_id_46>",
50
+ "<extra_id_47>",
51
+ "<extra_id_48>",
52
+ "<extra_id_49>",
53
+ "<extra_id_50>",
54
+ "<extra_id_51>",
55
+ "<extra_id_52>",
56
+ "<extra_id_53>",
57
+ "<extra_id_54>",
58
+ "<extra_id_55>",
59
+ "<extra_id_56>",
60
+ "<extra_id_57>",
61
+ "<extra_id_58>",
62
+ "<extra_id_59>",
63
+ "<extra_id_60>",
64
+ "<extra_id_61>",
65
+ "<extra_id_62>",
66
+ "<extra_id_63>",
67
+ "<extra_id_64>",
68
+ "<extra_id_65>",
69
+ "<extra_id_66>",
70
+ "<extra_id_67>",
71
+ "<extra_id_68>",
72
+ "<extra_id_69>",
73
+ "<extra_id_70>",
74
+ "<extra_id_71>",
75
+ "<extra_id_72>",
76
+ "<extra_id_73>",
77
+ "<extra_id_74>",
78
+ "<extra_id_75>",
79
+ "<extra_id_76>",
80
+ "<extra_id_77>",
81
+ "<extra_id_78>",
82
+ "<extra_id_79>",
83
+ "<extra_id_80>",
84
+ "<extra_id_81>",
85
+ "<extra_id_82>",
86
+ "<extra_id_83>",
87
+ "<extra_id_84>",
88
+ "<extra_id_85>",
89
+ "<extra_id_86>",
90
+ "<extra_id_87>",
91
+ "<extra_id_88>",
92
+ "<extra_id_89>",
93
+ "<extra_id_90>",
94
+ "<extra_id_91>",
95
+ "<extra_id_92>",
96
+ "<extra_id_93>",
97
+ "<extra_id_94>",
98
+ "<extra_id_95>",
99
+ "<extra_id_96>",
100
+ "<extra_id_97>",
101
+ "<extra_id_98>",
102
+ "<extra_id_99>"
103
+ ],
104
+ "eos_token": "</s>",
105
+ "pad_token": "<pad>",
106
+ "unk_token": "<unk>"
107
+ }
spiece.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:caa6e2f21aeec181276ab80273e3f869ce303ccb8602d68e0524783c3581092d
3
+ size 800223
spiece.vocab ADDED
The diff for this file is too large to render. See raw diff
 
test_results.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "test_bp": 1.0,
3
+ "test_precision_ng1": 73.84971296375632,
4
+ "test_precision_ng2": 51.14788183314563,
5
+ "test_precision_ng3": 38.4143670608848,
6
+ "test_precision_ng4": 29.72446177017808,
7
+ "test_ref_len": 11637,
8
+ "test_score": 45.5717846591585,
9
+ "test_sys_len": 11671
10
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,113 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<extra_id_0>",
4
+ "<extra_id_1>",
5
+ "<extra_id_2>",
6
+ "<extra_id_3>",
7
+ "<extra_id_4>",
8
+ "<extra_id_5>",
9
+ "<extra_id_6>",
10
+ "<extra_id_7>",
11
+ "<extra_id_8>",
12
+ "<extra_id_9>",
13
+ "<extra_id_10>",
14
+ "<extra_id_11>",
15
+ "<extra_id_12>",
16
+ "<extra_id_13>",
17
+ "<extra_id_14>",
18
+ "<extra_id_15>",
19
+ "<extra_id_16>",
20
+ "<extra_id_17>",
21
+ "<extra_id_18>",
22
+ "<extra_id_19>",
23
+ "<extra_id_20>",
24
+ "<extra_id_21>",
25
+ "<extra_id_22>",
26
+ "<extra_id_23>",
27
+ "<extra_id_24>",
28
+ "<extra_id_25>",
29
+ "<extra_id_26>",
30
+ "<extra_id_27>",
31
+ "<extra_id_28>",
32
+ "<extra_id_29>",
33
+ "<extra_id_30>",
34
+ "<extra_id_31>",
35
+ "<extra_id_32>",
36
+ "<extra_id_33>",
37
+ "<extra_id_34>",
38
+ "<extra_id_35>",
39
+ "<extra_id_36>",
40
+ "<extra_id_37>",
41
+ "<extra_id_38>",
42
+ "<extra_id_39>",
43
+ "<extra_id_40>",
44
+ "<extra_id_41>",
45
+ "<extra_id_42>",
46
+ "<extra_id_43>",
47
+ "<extra_id_44>",
48
+ "<extra_id_45>",
49
+ "<extra_id_46>",
50
+ "<extra_id_47>",
51
+ "<extra_id_48>",
52
+ "<extra_id_49>",
53
+ "<extra_id_50>",
54
+ "<extra_id_51>",
55
+ "<extra_id_52>",
56
+ "<extra_id_53>",
57
+ "<extra_id_54>",
58
+ "<extra_id_55>",
59
+ "<extra_id_56>",
60
+ "<extra_id_57>",
61
+ "<extra_id_58>",
62
+ "<extra_id_59>",
63
+ "<extra_id_60>",
64
+ "<extra_id_61>",
65
+ "<extra_id_62>",
66
+ "<extra_id_63>",
67
+ "<extra_id_64>",
68
+ "<extra_id_65>",
69
+ "<extra_id_66>",
70
+ "<extra_id_67>",
71
+ "<extra_id_68>",
72
+ "<extra_id_69>",
73
+ "<extra_id_70>",
74
+ "<extra_id_71>",
75
+ "<extra_id_72>",
76
+ "<extra_id_73>",
77
+ "<extra_id_74>",
78
+ "<extra_id_75>",
79
+ "<extra_id_76>",
80
+ "<extra_id_77>",
81
+ "<extra_id_78>",
82
+ "<extra_id_79>",
83
+ "<extra_id_80>",
84
+ "<extra_id_81>",
85
+ "<extra_id_82>",
86
+ "<extra_id_83>",
87
+ "<extra_id_84>",
88
+ "<extra_id_85>",
89
+ "<extra_id_86>",
90
+ "<extra_id_87>",
91
+ "<extra_id_88>",
92
+ "<extra_id_89>",
93
+ "<extra_id_90>",
94
+ "<extra_id_91>",
95
+ "<extra_id_92>",
96
+ "<extra_id_93>",
97
+ "<extra_id_94>",
98
+ "<extra_id_95>",
99
+ "<extra_id_96>",
100
+ "<extra_id_97>",
101
+ "<extra_id_98>",
102
+ "<extra_id_99>"
103
+ ],
104
+ "eos_token": "</s>",
105
+ "extra_ids": 100,
106
+ "name_or_path": "yhavinga/ul2-large-dutch",
107
+ "pad_token": "<pad>",
108
+ "sp_model_kwargs": {},
109
+ "special_tokens_map_file": null,
110
+ "tokenizer_class": "T5Tokenizer",
111
+ "unk_token": "<unk>",
112
+ "use_fast_tokenizer": false
113
+ }
training_state.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"step": 1241629}