System Description for Transperfect (2021)

Abstract: In this paper, we describe our participation in the 2021 Workshop on Asian Translation (team ID: tpt_wat). We submitted results for all six directions of the JPC2 patent task. As a first-time participant in the task, we attempted to identify a single configuration that provided the best overall results across all language pairs. All our submissions were created using single base transformer models, trained on only the task-specific data, using a consistent configuration of hyperparameters. In contrast to the uniformity of our methods, our results vary widely across the six language pairs.

## introduction the field of machine translation has seen rapid innovation in the last few years, with new model architectures, pre-training regimens, and computational algorithms emerging at a dizzying pace. however, translation of these techniques into industry practice occurs more slowly. companies utilizing these techniques must take into account considerations such as deployment costs (model speed and size), scalability, explainability, the complexity of training regimens (resource constraints limiting independent hyperparameter optimization for all language pairs), and risk management, against which advances yielding performance gains must be weighed. for our participation in the 2021 workshop on asian translation shared task on patent translation, we have applied a single, standardized data preparation and model training pipeline as a way of benchmarking the performance of this process. we conducted limited experiments to test different parameters, before 1 http://lotus.kuee.kyoto-u.ac.jp/wat/patent/ settling on the approach which provided the best overall results across all language pairs. our nmt systems are standard base transformer @xcite models, which were trained using only the data resources provided by the task organizers. these models used shared subword vocabularies created with sentencepiece @xcite . in contrast to the uniformity of our methods, our results varied widely across the six language pairs. different scoring metrics prevent the direct comparison of scores from different language pairs, but relative to the top performing model in each language pair, our scores ranged from 98.84% of the top score for the @xmath0, to 83.89% of the top score for @xmath1. below, we describe in detail our system architecture, hyperparameter configuration, hardware resources, and results. ## results our results show that for most language pairs, a shared vocabulary of size 8,000 achieved the best performance. for the @xmath2, using a vocabulary size of 32,000 produced better results. using a split vocabulary for these language pairs also resulted in better performance, whereas a shared vocabulary was advantageous for all other language pairs. in all cases, the inclusion of back translated training data resulted in higher validation scores. ## discussion in this shared task, we set out to identify a single configuration of hyperparameters that provided the best overall performance across all six language pairs. while this approach precluded the possibility of obtaining optimal performance for all language pairs, it afforded the opportunity to investigate which hyperparameters have similar effects on different language pairs, and which have varied effects on different language pairs. as different language pairs require different hyperparameters, any parameter that can be held fixed during the experimentation stage can create significant savings for companies training their own machine translation models. for instance, variation in parameters such as learning rate, dropout, embedding dimensions, and tying the weights of the source and target embedding layers seemed to have similar effects on performance across all language pairs that we tested. using back translated data to augment the training sets also appeared to be universally beneficial. however, the size of the vocabulary seemed to have quite different effects in different language pairs. we are not aware of any theoretical framework for explaining how the various hyperparameters interact to produce such different results, nor do we know of any way of predicting the optimal hyperparameters for a given language pair other than iterative experimentation. if additional resources are used, several additional steps have also been shown to be effective at boosting performance, but were not employed in these experiments in order to maintain maximum simplicity. these additional steps include using an ensemble of models for decoding, using larger model sizes, performing word segmentation prior to creating the vocabularies, ordering the training data using the output of a language model (a technique referred to as curriculum learning), and employing an additional model for right-to-left re-ranking. with minimal manual intervention, our models achieved results ranging from fair to excellent. the large variance in the relative performance of these systems shows that no "onesize-fits-all" yet exists for the problem of machine translation. despite monumental advances in the field over the past several years, achieving optimal performance requires careful selection of hyperparameters, and different configurations are required for different languages.