Low-Resource Translation as Language Modeling (2020)

Abstract: We present our submission to the very low resource supervised machine translation task at WMT20. We use a decoder-only transformer architecture and formulate the translation task as language modeling. To address the low-resource aspect of the problem, we pretrain over a similar language parallel corpus. Then, we employ an intermediate back-translation step before fine-tuning. Finally, we present an analysis of the system’s performance.

## introduction this work describes our system for translating in both directions between german (de) and the lowresource language upper sorbian (hsb). german is a widespread language with tens of millions of speakers; upper sorbian is a west slavic language spoken in germany, and it is recognized as an endangered language by @xcite . this system constitutes our submission to the shared task on very-low-resource supervised machine translation at wmt20. 1 the ultimate goal of the task is to translate a blind test set from upper sorbian into german and vice versa. the task is constrained, meaning that all data sets used for training are selected from a set of corpora provided by the organizers. our primary contribution is our application of a decoder-only language-modeling architecture to a low-resource translation task, which to our knowledge is not well-investigated. in sections 2 and 3, we discuss related work and our system itself. sections 4, 5, and 6 describe our architecture. sections 7 and 8 contain our results and analysis. in the case of low-resource languages, where there is an absence of adequately sized parallel corpora, recent techniques focus on transfer learning @xcite , relying on monolingual corpora @xcite , enriching the input to the system @xcite , or expanding it through back-translation @xcite . techniques related to back-translation include pseudo-labeling and self-labeling. pseudo-labeling uses partially accurate data for training @xcite generated from knowledge bases, heuristic functions and crowdsourcing. selflabeling is an area that lies between self-supervised learning and pseudo-labeling @xcite . the model is used to predict labels for an unlabeled dataset and then is trained on this dataset. ## overview our system uses a transformer architecture, though instead of the traditional encoder-decoder layout, we use a single decoder-only transformer as do @xcite , formulating the translation task as a language modeling task. this architecture was suggested by @xcite and explored concretely by @xcite for widely-used languages. unlike previous approaches, in this method there is no intermediate representation of the input; instead, the translation is predicted directly through the attention mechanism. furthermore, in our submission, we rely on a similar-language pretraining task with a shared vocabulary, using czech (cs) / english (en) sentence pairs, similarly to @xcite and @xcite . finally, we supplement these techniques with traditional back-translation, using monolingual corpora in the target languages. ## training method figure 2 shows the training method. in total, our method consists of five individual steps. these include: a pretraining step, an intermediate step made up of three sub-steps (a pre-fine-tuning step, back-translation, back-translated training), and a final fine-tuning step. the following subsections describe these steps in detail. all of these steps (except the back-translation step, which is performed in inference mode) are performed as translation tasks using parallel corpora. the parallel corpora are either real or synthetic (in the case of the corpora resulting from back-translation). ## implementation we follow the gpt2 paper @xcite for the model architecture, excepting hyperparameters. an overview of the system hyperparameters is shown in we used a single nvidia gtx 1080ti gpu during training, and training times are shown in figure 2 . we argue that our method is time and resource efficient, easy to reproduce, and powerful. ## evaluation in this section, we provide details about our implementation and the final results of the submitted system on the shared task: translation between german and upper sorbian. ## conclusion since our model produces high-quality translations, we have shown that a small decoder-only transformer, configured to perform classical language modeling, is an effective translation system for lowresource language pairs. furthermore, we have shown that a similar language translation pretraining task can contribute substantially to the quality of such translation systems. finally, we have provided an analysis of the model's components and their relative contribution to its ultimate performance. further investigation would be needed to understand our model's relationship to other architectures under the same data sets and pretraining tasks.