initial commit

README.md

@@ -0,0 +1,69 @@
+---
+language:
+- en
+- nb
+- nn
+
+tags:
+- translation
+
+license: apache-2.0
+---
+
+### eng-nor
+
+* source language name: English
+* target language name: Norwegian
+* OPUS readme: [README.md](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-nor/README.md)
+
+* model: transformer-align
+* source language code: en
+* target language codes: nb, nn
+* dataset: opus  with backtranslations
+* release date: 2021-04-20
+* pre-processing: normalization + SentencePiece (spm32k,spm32k)
+* download original weights: [opus+bt-2021-04-20.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-nor/opus+bt-2021-04-20.zip/eng-nor/opus+bt-2021-04-20.zip)
+* a sentence-initial language token is required in the form of >>id<<(id = valid, usually three-letter target language ID)
+* Training data: 
+  * eng-nno: Tatoeba-train (1661769) wikipedia.aa.nno-eng (995603) wikipedia.ab.nno-eng (605107) wikiquote.aa.nno-eng (22626)
+  * eng-nob: Tatoeba-train (11525999) wikibooks.aa.nob-eng (37901) wikinews.aa.nob-eng (8706) wikipedia.aa.nob-eng (992563) wikipedia.ab.nob-eng (992772) wikipedia.ac.nob-eng (992621) wikipedia.ad.nob-eng (992828) wikipedia.ae.nob-eng (992812) wikipedia.af.nob-eng (976715) wikiquote.aa.nob-eng (10443) wikisource.aa.nob-eng (279891)
+* Validation data: 
+  * eng-nno: Tatoeba-dev, 505
+  * eng-nob: Tatoeba-dev, 5189
+  * total-size-shuffled: 1505
+  * devset-selected: top 1505  lines of Tatoeba-dev.src.shuffled
+* Test data: 
+  * Tatoeba-test.eng-nno: 460/3428
+  * Tatoeba-test.eng-nob: 4539/36110
+  * Tatoeba-test.eng-nor: 4999/39547
+* test set translations file: [test.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-nor/opus+bt-2021-04-20.zip/eng-nor/opus+bt-2021-04-20.test.txt)
+* test set scores file: [eval.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-nor/opus+bt-2021-04-20.zip/eng-nor/opus+bt-2021-04-20.eval.txt)
+* BLEU-scores
+|Test set|score|
+|---|---|
+|Tatoeba-test.eng-nob|56.4|
+|Tatoeba-test.eng-nor|55.4|
+|Tatoeba-test.eng-nno|40.3|
+* chr-F-scores
+|Test set|score|
+|---|---|
+|Tatoeba-test.eng-nob|0.716|
+|Tatoeba-test.eng-nor|0.71|
+|Tatoeba-test.eng-nno|0.615|
+
+### System Info: 
+* hf_name: eng-nor
+* source_languages: en
+* target_languages: nb,nn
+* opus_readme_url: https://object.pouta.csc.fi/Tatoeba-MT-models/eng-nor/opus+bt-2021-04-20.zip/README.md
+* original_repo: Tatoeba-Challenge
+* tags: ['translation']
+* languages: ['en', 'nb', 'nn']
+* src_constituents: ['eng']
+* tgt_constituents: ['nob', 'nno']
+* src_multilingual: False
+* tgt_multilingual: True
+* helsinki_git_sha: 59400fea592520766f9910390155681bc930dbc4
+* transformers_git_sha: fd5cdaeea6eafac32e9d967327bfa3dc0e0d962d
+* port_machine: DESKTOP-6CPR2HH
+* port_time: 2023-01-23-21:07

config.json

@@ -0,0 +1,44 @@
+{
+  "activation_dropout": 0.0,
+  "activation_function": "swish",
+  "architectures": [
+    "MarianMTModel"
+  ],
+  "attention_dropout": 0.0,
+  "bad_words_ids": [
+    [
+      54214
+    ]
+  ],
+  "bos_token_id": 0,
+  "d_model": 512,
+  "decoder_attention_heads": 8,
+  "decoder_ffn_dim": 2048,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 6,
+  "decoder_start_token_id": 54214,
+  "decoder_vocab_size": 54215,
+  "dropout": 0.1,
+  "encoder_attention_heads": 8,
+  "encoder_ffn_dim": 2048,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 6,
+  "eos_token_id": 40041,
+  "forced_eos_token_id": 40041,
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "max_length": 512,
+  "max_position_embeddings": 512,
+  "model_type": "marian",
+  "normalize_embedding": false,
+  "num_beams": 6,
+  "num_hidden_layers": 6,
+  "pad_token_id": 54214,
+  "scale_embedding": true,
+  "share_encoder_decoder_embeddings": true,
+  "static_position_embeddings": true,
+  "torch_dtype": "float16",
+  "transformers_version": "4.26.0.dev0",
+  "use_cache": true,
+  "vocab_size": 54215
+}

generation_config.json

@@ -0,0 +1,16 @@
+{
+  "_from_model_config": true,
+  "bad_words_ids": [
+    [
+      54214
+    ]
+  ],
+  "bos_token_id": 0,
+  "decoder_start_token_id": 54214,
+  "eos_token_id": 40041,
+  "forced_eos_token_id": 40041,
+  "max_length": 512,
+  "num_beams": 6,
+  "pad_token_id": 54214,
+  "transformers_version": "4.26.0.dev0"
+}

metadata.json

@@ -0,0 +1 @@
+{"hf_name": "eng-nor", "source_languages": "en", "target_languages": "nb,nn", "opus_readme_url": "https://object.pouta.csc.fi/Tatoeba-MT-models/eng-nor/opus+bt-2021-04-20.zip/README.md", "original_repo": "Tatoeba-Challenge", "tags": ["translation"], "languages": ["en", "nb", "nn"], "src_constituents": ["eng"], "tgt_constituents": ["nob", "nno"], "src_multilingual": false, "tgt_multilingual": true, "helsinki_git_sha": "59400fea592520766f9910390155681bc930dbc4", "transformers_git_sha": "fd5cdaeea6eafac32e9d967327bfa3dc0e0d962d", "port_machine": "DESKTOP-6CPR2HH", "port_time": "2023-01-23-21:07"}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8daa0ddf104081a4e798e59aa451b152525f48efc43aae55f6fea2ca959e5f82
+size 199500873

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "eos_token": "</s>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "eos_token": "</s>",
+  "model_max_length": 512,
+  "name_or_path": "marian_ckpt/eng-nor",
+  "pad_token": "<pad>",
+  "separate_vocabs": false,
+  "source_lang": "eng",
+  "sp_model_kwargs": {},
+  "special_tokens_map_file": null,
+  "target_lang": "nor",
+  "tokenizer_class": "MarianTokenizer",
+  "unk_token": "<unk>"
+}