AIST AIRC Submissions to the WMT 23 Shared Task (2023)

Abstract: This paper describes the development process of NMT systems that were submitted to the WMT 2023 General Translation task by the team of AIST AIRC. We trained constrained track models for translation between English, German, and Japanese. Before training the final models, we first filtered the parallel and monolingual data, then performed iterative back-translation as well as parallel data distillation to be used for non-autoregressive model training. We experimented with training Transformer models, Mega models, and custom non-autoregressive sequence-to-sequence models with encoder and decoder weights initialised by a multilingual BERT base. Our primary submissions contain translations from ensembles of two Mega model checkpoints and our contrastive submissions are generated by our non-autoregressive models.

## introduction we describe the machine translation (mt) systems submitted to the wmt 2023 general translation task developed by the team of aist airc. we experimented with data quality control by carefully filtering out noisy examples from parallel and monolingual data sets before training, and corpora selection by holding out specific web-crawled data. we also compared several modelling approaches by contrasting the well-known transformer architecture @xcite to several more recent ones, such as the mega model @xcite , as well as our own custom implementation of a non-autoregressive model with the encoder and decoder initialised by bert checkpoints. during the shared task submission week another new efficient architecture was published -the retentive network (retnet; @xcite , which we include in the paper as an ablation study. our main findings are: 1) non-autoregressive models can reach comparable output quality to the best autoregressive models while improving inference latency up to 9x; 2) modern efficient autoregressive models like retnet and mega not only slightly outperform the transformer in latency, but also in output quality; and 3) models trained on sentence-level data struggle to translate whole paragraphs -splitting them into sentences helps a lot, especially for the non-autoregressive model. ## data we only participated in the constrained track of the shared task; therefore, we limited our data set use to only the corpora provided by the shared task organisers. in specific experimentation configurations, we chose to leave out web-crawled data such as paracrawl and wikimatrix, but eventually kept them in our final submissions. all parallel training data and monolingual data for back-translation were filtered before starting any training, which has been proven very effective in previous wmt shared tasks @xcite and detailed by @xcite . parallel data distillation was performed only for training the nonautoregressive models, while for all autoregressive models, we used only pure clean parallel data. for the system development process, we selected news test sets from previous older wmt shared tasks as development data and the most recent ones as evaluation data. full statistics of the data we used are shown in ## model configurations while it is often possible to train ever larger models on more data requiring infinitely growing amounts of compute power which later become costly to deploy, we decided to approach our selection from the perspective of limiting environmental impact. in our pursuit of the final submission, we aimed to explore several modelling approaches with efficient decoding while still striving to maintain or improve output quality. for this we chose the baseline transformer model as our baseline, the recently introduced mega model @xcite , a custom implementation of a non-autoregressive model with bert-initialised encoder and decoder, and as an ablation study trained after the shared task submission deadline -retnet @xcite . each model was trained on a single machine with four nvidia v100 (16gb) gpus until convergence on development data (no improvement on validation loss for 7 checkpoints). the total trainable parameter counts for the four models are as follows: transformer -73,886,208; retnet -77,930,496; mega -63,367,854; bnb -384,214,027. ## results tables 3 and 4 list the progression of our different modelling methods and data selection approaches. we first started with training the transformer models as our baselines using only non-web-crawled parallel training data and compared it to mega models trained on the same data, while the larger paracrawl corpora were still filtering. initial results suggested that the transformer model optimises towards the development data slightly too much while ending up strongly outperformed by the mega model on evaluation data. from there on, we opted for using mega as our main model, and experimented with adding filtered paracrawl data to the training mix, which improved translation quality for all directions. we then used these four models (with paracrawl column in as an ablation study of adding another efficient model baseline, after the submission week had ended we trained retnet models, which were published on arxiv along with code on github during the submission week. ## conclusion in this paper we described the development process of the aist airc's nmt systems that were submitted for the wmt 2023 shared task on general domain text translation. we compared transformer models to mega, retnet and bert-nar-bert model architectures in search of efficient decoding approaches while still improving upon output quality. we showed that the transformer models can be outperformed by mega and ret-net in both translation quality, as well as inference speed, while bnb remained fastest in inference, but still lowest in quality. we also found that even though modern models should be able to handle long sequences, splitting the english↔german document-level data into separate sentences, translating and recombining them yielded better results. this should, however, be mitigable by training dedicated document-level models with appropriate training data. in total, output from four systems was submitted to the shared taks by airc for the english↔german and english↔japanese language pairs in both translation directions. in future work, we plan to experiment with replacing the bert models in bnb with other more efficient pre-trained language models which can be used as encoders/decoders, as well as incorporating document-level training data and modelling longer sequences with available data. in terms of data, we intend to increase vocabulary coverage by adding all known unicode emoji symbols to the vocabulary even if they are not present in the training data, as well as additionally sample paracrawl data where emoji are present.