Spaces:
Runtime error
Runtime error
| # Jointly Learning to Align and Translate with Transformer Models (Garg et al., 2019) | |
| This page includes instructions for training models described in [Jointly Learning to Align and Translate with Transformer Models (Garg et al., 2019)](https://arxiv.org/abs/1909.02074). | |
| ## Training a joint alignment-translation model on WMT'18 En-De | |
| ##### 1. Extract and preprocess the WMT'18 En-De data | |
| ```bash | |
| ./prepare-wmt18en2de_no_norm_no_escape_no_agressive.sh | |
| ``` | |
| ##### 2. Generate alignments from statistical alignment toolkits e.g. Giza++/FastAlign. | |
| In this example, we use FastAlign. | |
| ```bash | |
| git clone git@github.com:clab/fast_align.git | |
| pushd fast_align | |
| mkdir build | |
| cd build | |
| cmake .. | |
| make | |
| popd | |
| ALIGN=fast_align/build/fast_align | |
| paste bpe.32k/train.en bpe.32k/train.de | awk -F '\t' '{print $1 " ||| " $2}' > bpe.32k/train.en-de | |
| $ALIGN -i bpe.32k/train.en-de -d -o -v > bpe.32k/train.align | |
| ``` | |
| ##### 3. Preprocess the dataset with the above generated alignments. | |
| ```bash | |
| fairseq-preprocess \ | |
| --source-lang en --target-lang de \ | |
| --trainpref bpe.32k/train \ | |
| --validpref bpe.32k/valid \ | |
| --testpref bpe.32k/test \ | |
| --align-suffix align \ | |
| --destdir binarized/ \ | |
| --joined-dictionary \ | |
| --workers 32 | |
| ``` | |
| ##### 4. Train a model | |
| ```bash | |
| fairseq-train \ | |
| binarized \ | |
| --arch transformer_wmt_en_de_big_align --share-all-embeddings \ | |
| --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --activation-fn relu\ | |
| --lr 0.0002 --lr-scheduler inverse_sqrt --warmup-updates 4000 --warmup-init-lr 1e-07 \ | |
| --dropout 0.3 --attention-dropout 0.1 --weight-decay 0.0 \ | |
| --max-tokens 3500 --label-smoothing 0.1 \ | |
| --save-dir ./checkpoints --log-interval 1000 --max-update 60000 \ | |
| --keep-interval-updates -1 --save-interval-updates 0 \ | |
| --load-alignments --criterion label_smoothed_cross_entropy_with_alignment \ | |
| --fp16 | |
| ``` | |
| Note that the `--fp16` flag requires you have CUDA 9.1 or greater and a Volta GPU or newer. | |
| If you want to train the above model with big batches (assuming your machine has 8 GPUs): | |
| - add `--update-freq 8` to simulate training on 8x8=64 GPUs | |
| - increase the learning rate; 0.0007 works well for big batches | |
| ##### 5. Evaluate and generate the alignments (BPE level) | |
| ```bash | |
| fairseq-generate \ | |
| binarized --gen-subset test --print-alignment \ | |
| --source-lang en --target-lang de \ | |
| --path checkpoints/checkpoint_best.pt --beam 5 --nbest 1 | |
| ``` | |
| ##### 6. Other resources. | |
| The code for: | |
| 1. preparing alignment test sets | |
| 2. converting BPE level alignments to token level alignments | |
| 3. symmetrizing bidirectional alignments | |
| 4. evaluating alignments using AER metric | |
| can be found [here](https://github.com/lilt/alignment-scripts) | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{garg2019jointly, | |
| title = {Jointly Learning to Align and Translate with Transformer Models}, | |
| author = {Garg, Sarthak and Peitz, Stephan and Nallasamy, Udhyakumar and Paulik, Matthias}, | |
| booktitle = {Conference on Empirical Methods in Natural Language Processing (EMNLP)}, | |
| address = {Hong Kong}, | |
| month = {November}, | |
| url = {https://arxiv.org/abs/1909.02074}, | |
| year = {2019}, | |
| } | |
| ``` | |