Cheating to Identify Hard Problems for Neural Machine Translation (2023)

Abstract: We identify hard problems for neural machine translation models by analyzing progressively higher-scoring translations generated by letting models cheat to various degrees. If a system cheats and still gets something wrong, that suggests it is a hard problem. We experiment with two forms of cheating: providing the model a compressed representation of the target as an additional input, and fine-tuning on the test set. Contrary to popular belief, we find that the most frequent tokens are not necessarily the most accurately translated due to these often being function words and punctuation that can be used more flexibly in translation, or content words which can easily be paraphrased. We systematically analyze system outputs to identify categories of tokens which are particularly hard for the model to translate, and find that this includes certain types of named entities, subordinating conjunctions, and unknown and foreign words. We also encounter a phenomenon where words, often names, which were not infrequent in the training data are still repeatedly mistranslated by the models — we dub this the Fleetwood Mac problem.

## introduction some types and components of text are more difficult to translate than others. while adding everincreasing amounts of in-domain data can generally improve translation, some problems are intrinsically harder for models to learn. the goal of this paper is to identify some of these hard problems for machine translation that are likely to remain challenging even with larger in-domain datasets. the way we approach this is to cheat. @xcite introduced a method to provide a highly compressed representation of the desired output (a "cheat code") as an auxiliary input to the model so that the produced output is pushed to be closer to the target output. while their work was motivated as a method to estimate the amount of information present in the target that is missing in the source, we adopt the same method to produce unrealistically accurate models, and contend that if the models get particular things wrong even with hints from cheat codes, those are the harder things to translate. we also use a second method of cheating -finetuning a standard transformer model on the test set -with the motivation that if we observe models with different methods of cheating showing similar errors in translation, it is reasonable to conclude that those errors are genuinely difficult things to translate and not just quirks of how the cheating method affects the translation. while large amounts of in-domain data can improve overall quality significantly @xcite , this fine-tuning method lets us expose the model to the most relevant data possible, the test set itself. the longer we fine-tune, the more it learns to cheat and becomes more accurate on the test set. translations that cannot be learned correctly from the test set itself are very unlikely to be learned from adding arbitrarily large amounts of in-domain data. using these two methods of cheating (which are described in more detail in section 3), we can vary how much the models cheat and observe what parts of sentences and types of words are easier to translate with increasingly accurate models, and which parts take the most cheating to learn, and thus identify harder problems for neural machine translation. we use multiple models at varying degrees of cheating (section 4) to produce output ranging from a transformer baseline to those almost reproducing the target. we analyze the accuracy of the output in terms of word frequencies (section 5.1), parts of speech (section 5.2), and named entities (section 5.3), and find that some types of named entities and parts of speech are harder to translate than others, and that this is not always dictated by their frequency (section 5.4). automatic machine translation evaluation metrics such as bleu @xcite , chrf @xcite , meteor @xcite , comet @xcite , and bleurt @xcite exist in abundance, but a more fine-grained view of the errors made by translation systems is often required to determine weaknesses of models. @xcite provided a framework for manual classification of errors from statistical machine translation systems, and @xcite , and @xcite presented automated alternatives to such time-and effort-consuming human analysis. @xcite presented a highlevel analysis of challenges for neural machine translation. there are also methods to evaluate specific aspects of machine translation, such as contrastive translations to evaluate pronoun translation @xcite , transliteration or morphosyntactic agreement @xcite , and challenge sets @xcite . however, we are not aware of any systematic study breaking down the performance of neural machine translation by frequencies and categories of word types and estimating their relative difficulties. phenomena such as rare words and named entities being inaccurately translated are considered common knowledge and numerous works @xcite @xcite have offered various solutions to the problem. subword segmentation @xcite is the most commonly used method to improve the translation of rare words, but @xcite 's analysis also showed that while it significantly improves the translation of conjugated and compound words, the models still struggle with names due to inconsistent segmentation and ambiguous transliteration. other methods such as using source-target token alignments to translate out-of-vocabulary words using a dictionary @xcite depend upon the presence of suitable dictionaries and can usually be used only in specific use cases. tools such as compare-mt @xcite ) and mt-telescope @xcite aggregate different kinds of analyses based on token frequencies, types of words, and linguistic labels (such as parts of speech or named entities) together into reports to provide a detailed view of the errors in machine translation output, which we use for our purposes. 3 cheating methods we use two methods of cheating for the purposes of our analysis, "cheat codes" and fine-tuning on the test set, which are described in this section. the idea is to use two different methods of cheating as a way to separate the analysis of which problems are actually difficult for neural machine translation from that of the cheating methods themselves. ## analysis we use compare-mt foot_2 @xcite to systematically analyze and compare the outputs of the different models. we use the normalize-punctuation.perl foot_3 script from moses @xcite to normalize punctuation on the target side before analysis. for partof-speech (pos) tagging and named entity recognition (ner) in english, we use the roberta-based @xcite ) en_core_web_trf foot_4 model from spacy. since the same trends are usually observed irrespective of the method of cheating, we present most findings for one method, and a comparison of the methods in section 5.5. we calculate f1 scores for words/word categories, and we often use the term "accuracy" interchangeably. ## limitations we believe this paper provides useful insight into machine translation quality and its challenges. however, there are some limitations to our analysis: while some findings such as named entities being hard to translate are likely to transfer to all language pairs, it is possible that some other results may vary for other language pairs due to the characteristics of the languages themselves. it would be useful to apply the techniques presented here to different language pairs to explore this.