Hugging Face's logo

kazandaev
/
mt5-base-en-ru

Text2Text Generation
Transformers
PyTorch
TensorFlow
JAX
TensorBoard
mt5
Generated from Trainer
Inference Endpoints
Model card Files Metrics Community
1
kazandaev commited on
Commit
e9a0891
·
1 Parent(s): 5eca7c3

Training in progress, step 5000

Browse files
Files changed (35) hide show
  1. .gitattributes +1 -0
  2. .gitignore +1 -0
  3. README.md +28 -0
  4. config.json +30 -0
  5. flax_model.msgpack +3 -0
  6. pytorch_model.bin +3 -0
  7. runs/Mar10_14-01-33_delta/1646920909.4174175/events.out.tfevents.1646920909.delta.321372.1 +3 -0
  8. runs/Mar10_14-01-33_delta/events.out.tfevents.1646920909.delta.321372.0 +3 -0
  9. runs/Mar10_14-02-49_delta/1646920976.9998732/events.out.tfevents.1646920977.delta.321372.3 +3 -0
  10. runs/Mar10_14-02-49_delta/events.out.tfevents.1646920976.delta.321372.2 +3 -0
  11. runs/Mar10_14-04-17_delta/1646921064.386428/events.out.tfevents.1646921064.delta.321372.5 +3 -0
  12. runs/Mar10_14-04-17_delta/events.out.tfevents.1646921064.delta.321372.4 +3 -0
  13. runs/Mar10_14-07-48_delta/1646921276.8583674/events.out.tfevents.1646921276.delta.322195.1 +3 -0
  14. runs/Mar10_14-07-48_delta/events.out.tfevents.1646921276.delta.322195.0 +3 -0
  15. runs/Mar10_14-17-30_delta/1646921862.1405847/events.out.tfevents.1646921862.delta.323621.1 +3 -0
  16. runs/Mar10_14-17-30_delta/1646922497.5454006/events.out.tfevents.1646922497.delta.323621.2 +3 -0
  17. runs/Mar10_14-17-30_delta/events.out.tfevents.1646921862.delta.323621.0 +3 -0
  18. runs/Mar10_14-28-04_delta/1646922512.673264/events.out.tfevents.1646922512.delta.323621.4 +3 -0
  19. runs/Mar10_14-28-04_delta/events.out.tfevents.1646922512.delta.323621.3 +3 -0
  20. runs/Mar10_14-29-11_delta/1646922560.4090955/events.out.tfevents.1646922560.delta.325313.1 +3 -0
  21. runs/Mar10_14-29-11_delta/events.out.tfevents.1646922560.delta.325313.0 +3 -0
  22. runs/Mar10_14-30-18_delta/1646922627.0140805/events.out.tfevents.1646922627.delta.325313.3 +3 -0
  23. runs/Mar10_14-30-18_delta/events.out.tfevents.1646922627.delta.325313.2 +3 -0
  24. runs/Mar10_14-38-51_delta/1646923147.9489207/events.out.tfevents.1646923147.delta.326407.1 +3 -0
  25. runs/Mar10_14-38-51_delta/events.out.tfevents.1646923147.delta.326407.0 +3 -0
  26. runs/Mar10_16-05-32_delta/1646928341.391632/events.out.tfevents.1646928341.delta.335038.1 +3 -0
  27. runs/Mar10_16-05-32_delta/events.out.tfevents.1646928341.delta.335038.0 +3 -0
  28. runs/Mar10_16-16-24_delta/1646929080.8949437/events.out.tfevents.1646929080.delta.336197.1 +3 -0
  29. runs/Mar10_16-16-24_delta/events.out.tfevents.1646929080.delta.336197.0 +3 -0
  30. special_tokens_map.json +1 -0
  31. spiece.model +3 -0
  32. tf_model.h5 +3 -0
  33. tokenizer.json +3 -0
  34. tokenizer_config.json +1 -0
  35. training_args.bin +3 -0
.gitattributes CHANGED
@@ -25,3 +25,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
25
  *.zip filter=lfs diff=lfs merge=lfs -text
26
  *.zstandard filter=lfs diff=lfs merge=lfs -text
27
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
25
  *.zip filter=lfs diff=lfs merge=lfs -text
26
  *.zstandard filter=lfs diff=lfs merge=lfs -text
27
  *tfevents* filter=lfs diff=lfs merge=lfs -text
28
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
.gitignore ADDED
@@ -0,0 +1 @@
 
 
1
+ checkpoint-*/
README.md ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: multilingual
3
+ datasets:
4
+ - mc4
5
+
6
+ license: apache-2.0
7
+ ---
8
+
9
+ [Google's mT5](https://github.com/google-research/multilingual-t5)
10
+
11
+ mT5 is pretrained on the [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) corpus, covering 101 languages:
12
+
13
+ Afrikaans, Albanian, Amharic, Arabic, Armenian, Azerbaijani, Basque, Belarusian, Bengali, Bulgarian, Burmese, Catalan, Cebuano, Chichewa, Chinese, Corsican, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Hindi, Hmong, Hungarian, Icelandic, Igbo, Indonesian, Irish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish, Kyrgyz, Lao, Latin, Latvian, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Nepali, Norwegian, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Samoan, Scottish Gaelic, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali, Sotho, Spanish, Sundanese, Swahili, Swedish, Tajik, Tamil, Telugu, Thai, Turkish, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, West Frisian, Xhosa, Yiddish, Yoruba, Zulu.
14
+
15
+ **Note**: mT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.
16
+
17
+ Pretraining Dataset: [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual)
18
+
19
+ Other Community Checkpoints: [here](https://huggingface.co/models?search=mt5)
20
+
21
+ Paper: [mT5: A massively multilingual pre-trained text-to-text transformer](https://arxiv.org/abs/2010.11934)
22
+
23
+ Authors: *Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel*
24
+
25
+
26
+ ## Abstract
27
+
28
+ The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We describe the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. All of the code and model checkpoints used in this work are publicly available.
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/home/alexey/work/ML/models/mt5-base-en-ru",
3
+ "architectures": [
4
+ "MT5ForConditionalGeneration"
5
+ ],
6
+ "d_ff": 2048,
7
+ "d_kv": 64,
8
+ "d_model": 768,
9
+ "decoder_start_token_id": 0,
10
+ "dropout_rate": 0.1,
11
+ "eos_token_id": 1,
12
+ "feed_forward_proj": "gated-gelu",
13
+ "initializer_factor": 1.0,
14
+ "is_encoder_decoder": true,
15
+ "layer_norm_epsilon": 1e-06,
16
+ "model_type": "mt5",
17
+ "num_decoder_layers": 12,
18
+ "num_heads": 12,
19
+ "num_layers": 12,
20
+ "output_attentions": true,
21
+ "output_past": true,
22
+ "pad_token_id": 0,
23
+ "relative_attention_num_buckets": 32,
24
+ "tie_word_embeddings": false,
25
+ "tokenizer_class": "T5Tokenizer",
26
+ "torch_dtype": "float32",
27
+ "transformers_version": "4.16.2",
28
+ "use_cache": true,
29
+ "vocab_size": 250112
30
+ }
flax_model.msgpack ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c870a08ac1a67aaf347c9078f6ec7b446ed135de73491161540cd5af8c05a7bf
3
+ size 2329617315
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5daea94c78f82ba0c8b1379cba23eec2e295bfdc535e603ef2994be9cf7c22b9
3
+ size 2329732045
runs/Mar10_14-01-33_delta/1646920909.4174175/events.out.tfevents.1646920909.delta.321372.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:117a77fa0907fc0dcd159b865e41fe1e2ec7981158a0afe8ee28b9beb0dccc11
3
+ size 5083
runs/Mar10_14-01-33_delta/events.out.tfevents.1646920909.delta.321372.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84d13e4a42041a80c6e823ced799b2b4700278cabcf5d775729d17257b3e8637
3
+ size 3601
runs/Mar10_14-02-49_delta/1646920976.9998732/events.out.tfevents.1646920977.delta.321372.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:288605f6753aa26fb5e3d8ef718f513d8b4fcca1a90ee6fba461120ebb233525
3
+ size 5083
runs/Mar10_14-02-49_delta/events.out.tfevents.1646920976.delta.321372.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d7a2f42a2b096e327213c6d5588962723918026681f3b5b7e3fd411d7722385e
3
+ size 3601
runs/Mar10_14-04-17_delta/1646921064.386428/events.out.tfevents.1646921064.delta.321372.5 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1be99013deed7a92a23a40af36a7461aa0069146056977bb42445f25774694ce
3
+ size 5083
runs/Mar10_14-04-17_delta/events.out.tfevents.1646921064.delta.321372.4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5ca750c29f2d2738305d68a178879002374da0db33aed9b8109f8516fb0778a
3
+ size 3601
runs/Mar10_14-07-48_delta/1646921276.8583674/events.out.tfevents.1646921276.delta.322195.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:83ce2bca6e866acdac909580e3762abe0fea6945ebd36f7dff9010c3a127babd
3
+ size 5083
runs/Mar10_14-07-48_delta/events.out.tfevents.1646921276.delta.322195.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:65b6b651db7ef1656ee84dd3348affc87a7cfa5e000892ea9a2aea175a769511
3
+ size 3600
runs/Mar10_14-17-30_delta/1646921862.1405847/events.out.tfevents.1646921862.delta.323621.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a4cf2ee2997fff788a80763625438d1f939adb09b580b58a0111f0285d39d80d
3
+ size 5083
runs/Mar10_14-17-30_delta/1646922497.5454006/events.out.tfevents.1646922497.delta.323621.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a7286efd24e0751120c0731bbda8c72b8e101ca24570cf39f4b21a4584167916
3
+ size 5083
runs/Mar10_14-17-30_delta/events.out.tfevents.1646921862.delta.323621.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f9d0ce405e9c75cc02f0ebe3e26aea8c68636f63206e5be1a772472ed20f2d1
3
+ size 3601
runs/Mar10_14-28-04_delta/1646922512.673264/events.out.tfevents.1646922512.delta.323621.4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98738e8aebc95a2d2ce4903296439e3604d71bc2b59c66e40172274ebc281314
3
+ size 5083
runs/Mar10_14-28-04_delta/events.out.tfevents.1646922512.delta.323621.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4c862aa9b73802e5e31f54ba9f5b77a64ba34b7959453004c4f5b33918fa245f
3
+ size 40
runs/Mar10_14-29-11_delta/1646922560.4090955/events.out.tfevents.1646922560.delta.325313.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9cd0a42ba4849a5645f813b3f0428184f4d85b53df161788e2a3aa79a12daba5
3
+ size 5083
runs/Mar10_14-29-11_delta/events.out.tfevents.1646922560.delta.325313.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af24223f421c94cf959201fcbd8becadfbb25ecb53ac1e48c8cc7e2b2afc6b7c
3
+ size 3601
runs/Mar10_14-30-18_delta/1646922627.0140805/events.out.tfevents.1646922627.delta.325313.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d79aa0a398fea618858c936489118976876ebc75d868c48b624839bd1434cb30
3
+ size 5083
runs/Mar10_14-30-18_delta/events.out.tfevents.1646922627.delta.325313.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f8769dd623954d2325bb4c32a33e6bfb9b5c72261465ee3c1db691e3ee1b75e0
3
+ size 3602
runs/Mar10_14-38-51_delta/1646923147.9489207/events.out.tfevents.1646923147.delta.326407.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c991367ad2fc4ce3ec423cc1a2f68535bdeaabcb34785f96f52443844632f14c
3
+ size 5083
runs/Mar10_14-38-51_delta/events.out.tfevents.1646923147.delta.326407.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8a4912762cbf4f9cef71b5601a8507cb06c6804c5b8c8e0ba8de234791affb78
3
+ size 4388
runs/Mar10_16-05-32_delta/1646928341.391632/events.out.tfevents.1646928341.delta.335038.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a1620df1a7e16903467124eeefb2e3ceb984075a2d0422bfa836fea00ba217ca
3
+ size 5083
runs/Mar10_16-05-32_delta/events.out.tfevents.1646928341.delta.335038.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e96744093b5777e2a195e5656e3c000f54804394cd563f55e1de2c4f7fe95984
3
+ size 3602
runs/Mar10_16-16-24_delta/1646929080.8949437/events.out.tfevents.1646929080.delta.336197.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d88eb201fb13dbc85e3d85fa02cb9f27eecacc5b20f3c7d3102dc3d588d8f987
3
+ size 5083
runs/Mar10_16-16-24_delta/events.out.tfevents.1646929080.delta.336197.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef42b0b0a0d200892aa9edb89617d46e7f26e032cae86ae83563f803a32499b9
3
+ size 5171
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}
spiece.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef78f86560d809067d12bac6c09f19a462cb3af3f54d2b8acbba26e1433125d6
3
+ size 4309802
tf_model.h5 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6162ca01474a959272fbf74b24d87b0c3fb7edb2099bdeef8953e96059c4e60b
3
+ size 2330172944
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dc41ffe02668685117e7b0788b03e4ca4fad824b252d1a2a2db13f9ece73854f
3
+ size 16330369
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>", "extra_ids": 0, "additional_special_tokens": null, "special_tokens_map_file": "/home/patrick/.cache/torch/transformers/685ac0ca8568ec593a48b61b0a3c272beee9bc194a3c7241d15dcadb5f875e53.f76030f3ec1b96a8199b2593390c610e76ca8028ef3d24680000619ffb646276", "name_or_path": "/home/alexey/work/ML/models/mt5-base-en-ru", "sp_model_kwargs": {}, "tokenizer_class": "T5Tokenizer"}
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba5ab1da1cef5eb9492c57a1ca4cdb622bd773fcc92702760254173389b3d0d5
3
+ size 3183