Spaces:
Build error
Build error
A newer version of the Gradio SDK is available:
4.42.0
Flores101: Large-Scale Multilingual Machine Translation
Introduction
Baseline pretrained models for small and large tracks of WMT 21 Large-Scale Multilingual Machine Translation competition.
Flores Task at WMT 21: http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html
Flores announement blog post: https://ai.facebook.com/blog/flores-researchers-kick-off-multilingual-translation-challenge-at-wmt-and-call-for-compute-grants/
Pretrained models
Model | Num layers | Embed dimension | FFN dimension | Vocab Size | #params | Download |
---|---|---|---|---|---|---|
flores101_mm100_615M |
12 | 1024 | 4096 | 256,000 | 615M | https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_615M.tar.gz |
flores101_mm100_175M |
6 | 512 | 2048 | 256,000 | 175M | https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_175M.tar.gz |
These models are trained similar to M2M-100 with additional support for the languages that are part of the WMT Large-Scale Multilingual Machine Translation track. Full list of languages can be found at the bottom.
Example Generation code
Download model, sentencepiece vocab
fairseq=/path/to/fairseq
cd $fairseq
# Download 615M param model.
wget https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_615M.tar.gz
# Extract
tar -xvzf flores101_mm100_615M.tar.gz
Encode using our SentencePiece Model
Note: Install SentencePiece from here
fairseq=/path/to/fairseq
cd $fairseq
# Download example dataset From German to French
sacrebleu --echo src -l de-fr -t wmt19 | head -n 20 > raw_input.de-fr.de
sacrebleu --echo ref -l de-fr -t wmt19 | head -n 20 > raw_input.de-fr.fr
for lang in de fr ; do
python scripts/spm_encode.py \
--model flores101_mm100_615M/sentencepiece.bpe.model \
--output_format=piece \
--inputs=raw_input.de-fr.${lang} \
--outputs=spm.de-fr.${lang}
done
Binarization
fairseq-preprocess \
--source-lang de --target-lang fr \
--testpref spm.de-fr \
--thresholdsrc 0 --thresholdtgt 0 \
--destdir data_bin \
--srcdict flores101_mm100_615M/dict.txt --tgtdict flores101_mm100_615M/dict.txt
Generation
fairseq-generate \
data_bin \
--batch-size 1 \
--path flores101_mm100_615M/model.pt \
--fixed-dictionary flores101_mm100_615M/dict.txt \
-s de -t fr \
--remove-bpe 'sentencepiece' \
--beam 5 \
--task translation_multi_simple_epoch \
--lang-pairs flores101_mm100_615M/language_pairs.txt \
--decoder-langtok --encoder-langtok src \
--gen-subset test \
--fp16 \
--dataset-impl mmap \
--distributed-world-size 1 --distributed-no-spawn
Supported Languages and lang code
Language | lang code |
---|---|
Akrikaans | af |
Amharic | am |
Arabic | ar |
Assamese | as |
Asturian | ast |
Aymara | ay |
Azerbaijani | az |
Bashkir | ba |
Belarusian | be |
Bulgarian | bg |
Bengali | bn |
Breton | br |
Bosnian | bs |
Catalan | ca |
Cebuano | ceb |
Chokwe | cjk |
Czech | cs |
Welsh | cy |
Danish | da |
German | de |
Dyula | dyu |
Greek | el |
English | en |
Spanish | es |
Estonian | et |
Persian | fa |
Fulah | ff |
Finnish | fi |
French | fr |
Western Frisian | fy |
Irish | ga |
Scottish Gaelic | gd |
Galician | gl |
Gujarati | gu |
Hausa | ha |
Hebrew | he |
Hindi | hi |
Croatian | hr |
Haitian Creole | ht |
Hungarian | hu |
Armenian | hy |
Indonesian | id |
Igbo | ig |
Iloko | ilo |
Icelandic | is |
Italian | it |
Japanese | ja |
Javanese | jv |
Georgian | ka |
Kachin | kac |
Kamba | kam |
Kabuverdianu | kea |
Kongo | kg |
Kazakh | kk |
Central Khmer | km |
Kimbundu | kmb |
Northern Kurdish | kmr |
Kannada | kn |
Korean | ko |
Kurdish | ku |
Kyrgyz | ky |
Luxembourgish | lb |
Ganda | lg |
Lingala | ln |
Lao | lo |
Lithuanian | lt |
Luo | luo |
Latvian | lv |
Malagasy | mg |
Maori | mi |
Macedonian | mk |
Malayalam | ml |
Mongolian | mn |
Marathi | mr |
Malay | ms |
Maltese | mt |
Burmese | my |
Nepali | ne |
Dutch | nl |
Norwegian | no |
Northern Sotho | ns |
Nyanja | ny |
Occitan | oc |
Oromo | om |
Oriya | or |
Punjabi | pa |
Polish | pl |
Pashto | ps |
Portuguese | pt |
Quechua | qu |
Romanian | ro |
Russian | ru |
Sindhi | sd |
Shan | shn |
Sinhala | si |
Slovak | sk |
Slovenian | sl |
Shona | sn |
Somali | so |
Albanian | sq |
Serbian | sr |
Swati | ss |
Sundanese | su |
Swedish | sv |
Swahili | sw |
Tamil | ta |
Telugu | te |
Tajik | tg |
Thai | th |
Tigrinya | ti |
Tagalog | tl |
Tswana | tn |
Turkish | tr |
Ukrainian | uk |
Umbundu | umb |
Urdu | ur |
Uzbek | uz |
Vietnamese | vi |
Wolof | wo |
Xhosa | xh |
Yiddish | yi |
Yoruba | yo |
Chinese | zh |
Zulu | zu |