Evaluating Structural Generalization in Neural Machine Translation (2024)

Abstract: Compositional generalization refers to the ability to generalize to novel combinations of previously observed words and syntactic structures.Since it is regarded as a desired property of neural models, recent work has assessed compositional generalization in machine translation as well as semantic parsing.However, previous evaluations with machine translation have focused mostly on lexical generalization (i.e., generalization to unseen combinations of known words).Thus, it remains unclear to what extent models can translate sentences that require structural generalization (i.e., generalization to different sorts of syntactic structures).To address this question, we construct SGET, a machine translation dataset covering various types of compositional generalization with control of words and sentence structures.We evaluate neural machine translation models on SGET and show that they struggle more in structural generalization than in lexical generalization.We also find different performance trends in semantic parsing and machine translation, which indicates the importance of evaluations across various tasks.

## introduction humans can understand and produce novel language expressions by combining familiar words and syntactic structures @xcite . this ability is known as compositional generalization, which has recently attracted much attention in the context of investigating the generalization ability of neural models @xcite . compositional generalization is divided into two categories: lexical generalization and structural generalization @xcite @xcite . lexical generalization is generalization to an unseen combination of familiar lexical items regardless of syntactic structures, whereas structural generalization is generalization to an unseen com-(a) english-japanese translation (b) syntactic trees figure 1 : how we test the compositional generalization abilities of neural models on sget. this pattern focuses on the structural generalization to an adjective modifying a subject noun, which is unseen in the training. existing studies have investigated compositional generalization mainly with semantic parsing @xcite , considering both lexical and structural generalization. on the other hand, evaluation with application tasks is also essential to thoroughly evaluate the compositional generalization abilities of neural models @xcite . among those application tasks, several studies have focused on machine translation @xcite @xcite , as it can be considered a mapping from a source language to a target one. the translation result by a model should indicate how the model handles the lexical and syntactic features because correct translation requires a model to reflect the lexical and syntactic features of the source sentence. however, these studies focused primarily on lexical generalization, and structural generalization has yet to be investigated due to the difficulty of precise control of structural gaps between source and target sentences. in this paper, we propose sget foot_0 (structural generalization benchmark based on english-japanese machine translation), a parallel dataset for english-japanese translation tasks covering both lexical and structural generalization. figure 1 shows an example of a structural gap between the training and generalization sets in sget. the training set includes sentences where the object is a noun phrase in which an adjective modifies a noun. if a model generalizes compositionally, then it should be able to translate sentences with a noun phrase of the same structure in the subject position. we adopt a rule-based method to construct sget so that we can control lexical items and syntactic structures and strictly evaluate the compositional generalization abilities of models. the method involves generating english sentences with probabilistic context-free grammar (pcfg) and creating japanese parallel translations with rulebased machine translation (rbmt). we design the experimental settings so that non-essential factors such as sentence length @xcite and naturalness do not affect the evaluation. we evaluate lstm @xcite , transformer @xcite , and llama 2 @xcite on sget and analyze their performances. we also compare the results with those of previous studies based on semantic parsing. our contributions are as follows. ## conclusion in this paper, we constructed sget for evaluating the compositional generalization abilities of neural models in english-japanese machine translation. we generated the dataset via a rule-based approach, which allowed us to control the lexical items and syntactic structures in the dataset and strictly evaluate the compositional generalization abilities of models. we then used the generated dataset to investigate the compositional generalization abilities of lstm, vanilla transformer, and llama 2. we found that all models struggled to generalize compositionally to unseen constituents overall, while they achieved near-perfect scores on the indistribution test set. the results also suggested that for the models, structural generalization is more challenging than lexical generalization. these findings regarding machine translation are consistent with the results of previous studies on semantic parsing. however, the models correctly filled the structural gap questioned in some generalization patterns in which the models performed poorly in semantic parsing, which might have been because of the fairer settings that we adopted and the difference between the two tasks. this underlines the importance of evaluating the compositional generalization abilities of models under diverse settings, not only in semantic parsing but also in other tasks such as machine translation. we also discovered that the naturalness of the sentences in sget had little influence on the performance of llama 2. this indicates that a synthetically generated benchmark is adequate for evaluating compositional generalization in a controlled manner. future work might focus on comprehensive evaluations across various languages and tasks. the rbmt method adopted in our approach originally handles english-chinese translation in addition to english-japanese @xcite , so targeting other languages, including chinese, should be feasible.