Evaluating Morphological Generalisation in Machine Translation by Distribution-Based Compositionality Assessment (2023)

Abstract: Compositional generalisation refers to the ability to understand and generate a potentially infinite number of novel meanings using a finite group of known primitives and a set of rules to combine them. The degree to which artificial neural networks can learn this ability is an open question. Recently, some evaluation methods and benchmarks have been proposed to test compositional generalisation, but not many have focused on the morphological level of language. We propose an application of the previously developed distribution-based compositionality assessment method to assess morphological generalisation in NLP tasks, such as machine translation or paraphrase detection. We demonstrate the use of our method by comparing translation systems with different BPE vocabulary sizes. The evaluation method we propose suggests that small vocabularies help with morphological generalisation in NMT.

## introduction natural languages usually adhere to the principle of compositionality, with the exception of idiomatic expressions. @xcite phrased this principle as "the meaning of a whole is a function of the meanings of the parts and of the way they are syntactically combined". deriving from this principle, compositional generalisation (cg) refers to the capacity to understand and generate a potentially infinite number of novel meanings using a finite group of known primitives and a set of rules of how to combine them. in the case of language, morphemes are combined into words and words in turn into phrases and sentences, using the syntactical rules of the language. neural networks have long been argued to lack the ability to generalise compositionally the way humans do @xcite . after the rapid improvement of neural nlp systems during the previous decade, this question has gained renewed interest. many new evaluation methods have been developed to assess whether the modern sequence-to-sequence (seq2seq) architectures such as transformers exhibit cg, since they certainly exhibit increasingly competent linguistic behaviour. for instance, in one of the seminal cg evaluation methods, called @xcite , a seq2seq system has seen certain natural language commands in training and needs to combine them in novel ways in testing. cg is a general capacity that can be seen as a desideratum in many nlp tasks, and in machine learning more generally. furthermore, cg is a multifaceted concept that can be, and should be, decomposed into narrower, more manageable aspects that can be tested separately @xcite . for example, nlp systems should be able to generalise compositionally both on the level of words and on the level of morphology. although many aspects of cg have recently been evaluated in nlp (an extensive review is offered by @xcite ), some aspects have remained without an evaluation method. we identify (see section 2) a lack of methods to evaluate compositional morphological generalisation using only natural, non-synthetic, data. to fill this gap, we propose an application of the distribution-based compositionality assessment (dbca) method @xcite (henceforth keysers) to generate adversarial data splits to evaluate morphological generalisation in nlp systems. specifically, we split natural language corpora while controlling the distributions of lemmas and morphological features (atoms in the terminology of keysers) on the one hand, and the distributions of the combinations of atoms (compounds, not to be confused with compound words) on the other hand. by requiring a low divergence between the atom distributions of the train and test sets, and a high divergence between the compound distributions, we can evaluate how well a system is able to generalise its morphological knowledge to unseen word forms. for example, if our corpus included as atoms the lemmas "cat" and "dog", and the morphological tags @xmath0, a low divergence between the atom distributions would mean that both the training and test sets included all four of the atoms, and a high compound divergence would mean that the sets include different combinations of them, for instance training set {cat, dogs} and test set {cats, dog}. our main contributions are the following: firstly, we describe an application of dbca to evaluate morphological generalisation in any nlp task in which the train and test data consist of sentences for which morphological tags are available. secondly, we demonstrate how by this method we can evaluate morphological generalisation in machine translation without manual test design. and thirdly, using our proposed method, we assess the effect of the source language bpe @xcite vocabulary size in finnish-english nmt performance, and conclude that a smaller vocabulary helps the nmt models in morphological generalisation. ## background in the broader field of machine learning, cg has been analysed in various domains besides that of natural language, such as visual question answering @xcite , visual reasoning @xcite and mathematics @xcite , but in this work we focus on natural lan-guage tasks. two reviews have recently been published about cg in nlp, of which @xcite focus on semantic parsing and the aforementioned @xcite (henceforth hupkes) take a broader view, reviewing generalisation in general, not only the compositional type. hupkes categorised nlp generalisation experiments along five dimensions, of which we discuss two here to motivate our work. the first is the type of generalisation along which the compositional type is distinguished from the morphological type. hupkes define compositionality as "the ability to systematically recombine previously learned elements to map new inputs made up from these elements to their correct output. in language, the inputs are 'forms' (e.g. phrases, sentences, larger pieces of discourse), and the output that they need to be mapped to is their meaning ...". in nmt, the translation works as a proxy to meaning, so that cg can be evaluated by evaluating the translation @xcite (other works that assess cg in nmt include @xcite ). hupkes contrast compositional with structural, including morphological, generalisation where an output space is not required but which focuses on generation of the correct forms. these definitions suggest a clear divide between the categories, which is understandable when analysing the literature: morphological generalisation, specifically inflection generation, has for decades been studied in psycholinguistics @xcite and computational linguistics @xcite @xcite . these studies do not address the question of how the different inflections are mapped to different meanings, hence they do not address compositional generalisation. however, inflections do bear meaning, of course, and so compositional morphological generalisation is an ability that humans possess, and nlp systems ought to be tested on. although hupkes do not categorise any experiments as assessing compositional morphological generalisation, there has been at least one that we think could be so categorised: burlot and yvon (2017) designed an nmt test suite in which a single morphological feature is modified in a source language sentence, creating a contrastive pair, and the translations of the contrastive sentences are inspected for a corresponding change in the target language. the other dimension of hupkes relevant to the motivation of our experiments is that of shift source: the shift between train and test sets could occur naturally (as in two natural corpora in different domains), it can be created by generating synthetic data, or an artificial partition of natural data can be obtained. most of the previous methods to assess compositional generalisation in nmt @xcite @xcite have synthetised data for the test sets. generating synthetic data has its benefits: any morphological form can occur in the data when it is generated, and a single morphological feature can be easily focused on and evaluated qualitatively as well as quantitatively. however, synthetic data has at least practical disadvantages, leaving aside the more theoretical question of how well the synthetic language approximates natural language, assuming the ultimate goal is systems that process natural language. in practice, synthetic test sets require manual design, which means it is difficult to come by a method to generate an unlimited number of synthetic sentences, or a method that could work in arbitrary languages. furthermore, when manually designing test suites to evaluate morphological generalisation, as burlot and yvon (2017) designed, the requirement for manual work restricts the number of morphological phenomena we have resources to test. the other option is to create artificial data splits of natural data. while natural data may be noisier and it might be more difficult to focus on a specific phenomenon of the language by this method, this method is easier to automate completely. furthermore, the method of automatically generating data splits that we present in the next section is also generalisable to other tasks (e.g. paraphrase detection) and any corpus of sentences. generating artificial data splits of natural data has previously been used to test cg in translation @xcite , as well as to assess the capacity to capture long-distance dependencies in translation @xcite , but not to assess morphological generalisation, as far as we are aware. (for a more general discussion of splitting data into non-random testing and training sets, see @xcite .) the method we describe in this paper is an application of the dbca method developed by keysers. since this method is generic and taskagnostic, it can be applied to any dataset for which it is possible to define atom and compound distributions. although it is easier to define these distributions for synthetic data, as in the cfq dataset described by keysers, it can also be applied to natural data, for example in semantic parsing @xcite . the next section describes how dbca can be used to assess morphological generalisation in any task where the training and testing corpora consist of natural language sentences. to determine the atom and compound distributions, we first need to obtain the lemmas and morphological tags of all words in the corpus, which we accomplish for finnish corpora using the turku neural parser pipeline @xcite . for the experiments presented in section 4, we use a corpus of 1m sentences. in practice, we do not have resources to control the distribution of all lemmas even in this relatively small corpus, so we need to select some subset of the lemmas that we include in our analysis. selecting the lemma subset could be done in many ways, but the following is a way we deemed reasonable. to limit the number of lemmas, we first filter out lemmas that do not appear in the list of 94110 finnish lemmas 2 or, since this list does not include proper names, in lists 3 of names atoms compounds desc. lemmas and morphological tags combinations of atoms e.g. tunturi, @xmath1, @xmath2, @xmath3, @xmath4, of mountains); tunturi|@xmath5, on mountain) for places, or lists of finnish and english given names. this way, the lemmas that are filtered out include most of the typos and other nonwords. then we rank the remaining lemmas by frequency in our corpus, and sample a fixed number of lemma occurrences from constant intervals in the ranked list of lemmas. specifically, we take 40000 lemma occurrences at intervals of 1000 lemma types in the list of lemmas. for our corpus of 1m sentences, this method subsamples the lemmas with frequency ranks of 1000-1033, 2000-2083, 3000-3174, and so on, so that there are fewer frequent lemma types than rare lemma types, but the total number of occurrences of each bucket is around 40k. lemmas that occur fewer than 10 times in the corpus are excluded. after the filtering, we have 8720 lemma types that occur about 390k times in total in our corpus of 1m sentences. we append the list of 48 morphological tags 4 (after filtering some that indicate uninteresting words such as 'typo' and 'abbr') that these lemmas appear with to the lemma list to complete our list of atoms. keysers weighted the compounds to "avoid double-counting compounds that are highly correlated with some of their super-compounds". the idea is to lessen the weight of those compounds that only or often occur as a part of one certain super-compound. we weight the compounds analogously, but use only two levels in our weighting, which makes the weighting simpler than in keysers: we consider the combinations of morphological tags as the lower level of compounds, and these combined with lemmas as the higher level. thus the motivation for weighting in our case is not to use those morphological tag combinations that only occur with some specific lemma. therefore, we look for the lemma with and finnish: https://tinyurl.com/3mn52ms6 https://tinyurl.com/mwjvaxkk 4 see https://universaldependencies.org/ docs/fi/feat/ for the list of finnish morphological tags. after the described filtering steps, we have 8322 atoms, which includes the lemmas and morphological tags. the atoms occur about 1.3m times in 273k sentences in our corpus of 1m sentences. there are 335 morphological tag combinations, which create about 69k unique word forms with the lemmas; i.e. we use 69k compounds in our analysis. these compounds occur 352k times in the corpus. calculating atom and compound divergences is done the same way as in keysers. namely, divergence d between distributions p and q is calculated using the chernoff coefficient c @xcite , with α = 0.5 for the atom divergence and α = 0.1 for the compound divergence. as described by keysers, α = 0.5 for the atom divergence "reflects the desire of making the atom distributions in train and test as similar as possible", and α = 0.1 for the compound divergence "reflects the intuition that it is more important whether a certain compound occurs in p (train) than whether the probabilities in p (train) and q (test) match exactly". since the chernoff coefficient is a similarity metric, the atom and compound divergences of a train set v and a test set w are: procedure 1 data division algorithm. once the divergences are defined, we can split a corpus of natural language sentences into training and testing sets with an arbitrary compound and atom divergence values. for this, we use a simple greedy algorithm, sketched in algorithm 1. for a maximum compound divergence split, the score is calculated as score(q, p ) = d c (q∥p ) -d a (q∥p ), and in general, for any desired compound divergence value c: in practice, we do not have resources to calculate the max @xmath6. instead, at each iteration we take a subset g ′ ⊂ g, say 1000 sentences, and calculate max @xmath7. as mentioned above, this method can be used for any corpus that consists of natural language sentences for which the morphological tags can be obtained. in the next section we use this method to assess morphological generalisation in machine translation. ## discussion and future work in section 3, we proposed an application of dbca to divide any corpus of sentences, for which morphological tags are available, into training and test sets with similar distributions of lemmas and morphological tags but contrasting distributions of word forms, in order to assess morphological generalisation. by this method, we can take a large proportion of the morphological phenomena of a selected language into consideration, in our experiments 335 different morphological categories that together with about 8k lemmas create 69k unique finnish word forms, and evaluate the effects of the contrasting train/test distributions of the word forms in machine translation. this enables a different, complementing type of assessment of morphological generalisation than previous synthetic 744 chrf2++ using only corpus-wide metrics such as bleu, as we used, does not discriminate between the morphological errors, which we are interested in, and other kinds of translation errors. in the terminology of burlot and yvon (2017), this holistic, document-level evaluation can be contrasted with analytic evaluation that focuses more specifically on difficulties in morphology. a trick that could enable a more analytic assessment of the translations of the unseen word forms would be to align the words in the source sentences with the words in the reference translations and the words in the predicted translations, and evaluate only the translations of the parts of the sentences that correspond to the unseen word forms. similar method has been used previously for example by @xcite ; @xcite . especially combined with this word-alignment trick, we could also make our evaluation more fine-grained (this concept also from burlot and yvon (2017)), that is, our evaluation could differentiate between different types of mistakes. since we have the morphological tags, we could sort the words by morphological category and compare the translation accuracies to look for any especially difficult categories for the translation models. to demonstrate the use of our proposed method, we compared nmt models with different bpe vocabulary sizes, since vocabulary size has been hypothesised to affect the capacity to model morphology in translation. besides vocabulary size, there are many other model design choices that have been proposed to help either in generalisation or in capturing morphological phenomena. tokenisation methods that are more linguistically motivated than bpe, such as the morfessor methods @xcite or lmvr @xcite , should help with morphological generalisation since the tokens produced by these methods approximate the linguistic morphemes more closely. factored nmt systems @xcite can cover more of the target side vocabulary than subword-based nmt systems, which can also help in modelling the morphology of the target language. we hope our evaluation method will help assessing alternative nmt methods, such as these, from the perspective of morphological generalisation. the dbca method is general, and could be applied to a wide variety of tasks and datasets. our application of dbca is more specific, but it still inherits some of the generality of the original method. our method is directly applicable to any machine learning task in which the dataset consists of sentences for which the morphological tags are available. in the future, we intend to extend our assessment of morphological generalisation to other languages, as well as to other nlp tasks, such as paraphrase detection. ## conclusion we proposed a method to assess morphological generalisation by distribution-based compositionality assessment. because this method is fully automated, it enables more comprehensive assessment of morphological generalisation than previously proposed synthetic benchmarks, in terms of the number of inflection types we can evaluate. we used our method to assess nmt models with different bpe vocabulary sizes and found that models with smaller vocabularies are better at morphological generalisation than those with larger vocabularies. lastly, we discussed the varied future directions that our generalisable method offers, such as assessing morphological generalisation in other nlp tasks besides nmt. aimen zerroug, mohit vaishnav, julien colin, sebastian musslick, and thomas serre. 2022. a benchmark for compositional visual reasoning. arxiv preprint arxiv:2206.05379.