Spaces:
Build error
Build error
<p align="center"> | |
<img src="flores_logo.png" width="500"> | |
</p> | |
# Flores101: Large-Scale Multilingual Machine Translation | |
## Introduction | |
Baseline pretrained models for small and large tracks of WMT 21 Large-Scale Multilingual Machine Translation competition. | |
Flores Task at WMT 21: http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html | |
Flores announement blog post: https://ai.facebook.com/blog/flores-researchers-kick-off-multilingual-translation-challenge-at-wmt-and-call-for-compute-grants/ | |
## Pretrained models | |
Model | Num layers | Embed dimension | FFN dimension| Vocab Size | #params | Download | |
---|---|---|---|---|---|--- | |
`flores101_mm100_615M` | 12 | 1024 | 4096 | 256,000 | 615M | https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_615M.tar.gz | |
`flores101_mm100_175M` | 6 | 512 | 2048 | 256,000 | 175M | https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_175M.tar.gz | |
These models are trained similar to [M2M-100](https://arxiv.org/abs/2010.11125) with additional support for the languages that are part of the WMT Large-Scale Multilingual Machine Translation track. Full list of languages can be found at the bottom. | |
## Example Generation code | |
### Download model, sentencepiece vocab | |
```bash | |
fairseq=/path/to/fairseq | |
cd $fairseq | |
# Download 615M param model. | |
wget https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_615M.tar.gz | |
# Extract | |
tar -xvzf flores101_mm100_615M.tar.gz | |
``` | |
### Encode using our SentencePiece Model | |
Note: Install SentencePiece from [here](https://github.com/google/sentencepiece) | |
```bash | |
fairseq=/path/to/fairseq | |
cd $fairseq | |
# Download example dataset From German to French | |
sacrebleu --echo src -l de-fr -t wmt19 | head -n 20 > raw_input.de-fr.de | |
sacrebleu --echo ref -l de-fr -t wmt19 | head -n 20 > raw_input.de-fr.fr | |
for lang in de fr ; do | |
python scripts/spm_encode.py \ | |
--model flores101_mm100_615M/sentencepiece.bpe.model \ | |
--output_format=piece \ | |
--inputs=raw_input.de-fr.${lang} \ | |
--outputs=spm.de-fr.${lang} | |
done | |
``` | |
### Binarization | |
```bash | |
fairseq-preprocess \ | |
--source-lang de --target-lang fr \ | |
--testpref spm.de-fr \ | |
--thresholdsrc 0 --thresholdtgt 0 \ | |
--destdir data_bin \ | |
--srcdict flores101_mm100_615M/dict.txt --tgtdict flores101_mm100_615M/dict.txt | |
``` | |
### Generation | |
```bash | |
fairseq-generate \ | |
data_bin \ | |
--batch-size 1 \ | |
--path flores101_mm100_615M/model.pt \ | |
--fixed-dictionary flores101_mm100_615M/dict.txt \ | |
-s de -t fr \ | |
--remove-bpe 'sentencepiece' \ | |
--beam 5 \ | |
--task translation_multi_simple_epoch \ | |
--lang-pairs flores101_mm100_615M/language_pairs.txt \ | |
--decoder-langtok --encoder-langtok src \ | |
--gen-subset test \ | |
--fp16 \ | |
--dataset-impl mmap \ | |
--distributed-world-size 1 --distributed-no-spawn | |
``` | |
### Supported Languages and lang code | |
Language | lang code | |
---|--- | |
Akrikaans | af | |
Amharic | am | |
Arabic | ar | |
Assamese | as | |
Asturian | ast | |
Aymara | ay | |
Azerbaijani | az | |
Bashkir | ba | |
Belarusian | be | |
Bulgarian | bg | |
Bengali | bn | |
Breton | br | |
Bosnian | bs | |
Catalan | ca | |
Cebuano | ceb | |
Chokwe | cjk | |
Czech | cs | |
Welsh | cy | |
Danish | da | |
German | de | |
Dyula| dyu | |
Greek | el | |
English | en | |
Spanish | es | |
Estonian | et | |
Persian | fa | |
Fulah | ff | |
Finnish | fi | |
French | fr | |
Western Frisian | fy | |
Irish | ga | |
Scottish Gaelic | gd | |
Galician | gl | |
Gujarati | gu | |
Hausa | ha | |
Hebrew | he | |
Hindi | hi | |
Croatian | hr | |
Haitian Creole | ht | |
Hungarian | hu | |
Armenian | hy | |
Indonesian | id | |
Igbo | ig | |
Iloko | ilo | |
Icelandic | is | |
Italian | it | |
Japanese | ja | |
Javanese | jv | |
Georgian | ka | |
Kachin | kac | |
Kamba | kam | |
Kabuverdianu | kea | |
Kongo | kg | |
Kazakh | kk | |
Central Khmer | km | |
Kimbundu | kmb | |
Northern Kurdish | kmr | |
Kannada | kn | |
Korean | ko | |
Kurdish | ku | |
Kyrgyz | ky | |
Luxembourgish | lb | |
Ganda | lg | |
Lingala | ln | |
Lao | lo | |
Lithuanian | lt | |
Luo | luo | |
Latvian | lv | |
Malagasy | mg | |
Maori | mi | |
Macedonian | mk | |
Malayalam | ml | |
Mongolian | mn | |
Marathi | mr | |
Malay | ms | |
Maltese | mt | |
Burmese | my | |
Nepali | ne | |
Dutch | nl | |
Norwegian | no | |
Northern Sotho | ns | |
Nyanja | ny | |
Occitan | oc | |
Oromo | om | |
Oriya | or | |
Punjabi | pa | |
Polish | pl | |
Pashto | ps | |
Portuguese | pt | |
Quechua | qu | |
Romanian | ro | |
Russian | ru | |
Sindhi | sd | |
Shan | shn | |
Sinhala | si | |
Slovak | sk | |
Slovenian | sl | |
Shona | sn | |
Somali | so | |
Albanian | sq | |
Serbian | sr | |
Swati | ss | |
Sundanese | su | |
Swedish | sv | |
Swahili | sw | |
Tamil | ta | |
Telugu | te | |
Tajik | tg | |
Thai | th | |
Tigrinya | ti | |
Tagalog | tl | |
Tswana | tn | |
Turkish | tr | |
Ukrainian | uk | |
Umbundu | umb | |
Urdu | ur | |
Uzbek | uz | |
Vietnamese | vi | |
Wolof | wo | |
Xhosa | xh | |
Yiddish | yi | |
Yoruba | yo | |
Chinese| zh | |
Zulu | zu | |