On using distribution-based compositionality assessment to evaluate compositional generalisation in machine translation (2023)

Abstract: Compositional generalisation (CG), in NLP and in machine learning more generally, has been assessed mostly using artificial datasets. It is important to develop benchmarks to assess CG also in real-world natural language tasks in order to understand the abilities and limitations of systems deployed in the wild. To this end, our GenBench Collaborative Benchmarking Task submission utilises the distribution-based compositionality assessment (DBCA) framework to split the Europarl translation corpus into a training and a test set in such a way that the test set requires compositional generalisation capacity. Specifically, the training and test sets have divergent distributions of dependency relations, testing NMT systems’ capability of translating dependencies that they have not been trained on. This is a fully-automated procedure to create natural language compositionality benchmarks, making it simple and inexpensive to apply it further to other datasets and languages. The code and data for the experiments is available at https://github.com/aalto-speech/dbca.

## introduction an often-used definition, by @xcite , of the concept of compositionality of language is that the "meaning of a whole is a function of the meanings of the parts and of the way they are syntactically combined". a more stringent definition adds that the composition of meaning is systematic: each part's meaning is the same in all the different sequences it appears in, the syntactical rules work the same way for different parts, and the same function determines meaning for different wholes @xcite @xcite . compositionality enables generalising to new meanings by combining familiar parts, and to understand each other language users need to employ common systematic rules. a number of benchmarks have been developed to assess systematic generalisation from different perspectives and in different nlp tasks. many of these consist of artificial data, such as the popular scan @xcite and cogs @xcite benchmarks. these artificial datasets are typically constructed to be highly systematic, to include straightforward syntactical rules. natural languages, however, have varied irregularities, idiomatic expressions, and other exceptions to the rules, which make composition of meaning much more complicated than in the case of the highly-regular artificial datasets. it's therefore important to assess whether nlp systems are able to generalise systematically also in the case of natural language, where systematic rules are obscured by exceptions @xcite . works that aim to assess systematic generalisation in natural language tasks often still synthesise some part of the dataset in order to create test examples where systematic generalisation is needed (for example @xcite and @xcite ). this enables precise testing of specific systematic rules, but comprehensive test suites that would assess the use of, for example, numerous syntactical rules can be arduous to synthesise. to sidestep the need to synthesise examples, a natural language dataset can be partitioned into training and test sets in such a way that the test set includes examples whose processing requires some systematic generalisation ability. a framework for partitioning data in this way was developed by @xcite , called distribution-based compositionality assessment, or dbca for short. the main idea of dbca is to control the distributions of atoms (primitive elements) and compounds (combinations of the atoms) to get approximately the same atom distributions but divergent compound distributions in the training and test sets. we utilise the dbca framework in our gen-bench collaborative benchmarking task submission, which consists of train-test splits of the europarl parallel corpus with divergent distributions sentence atoms compounds "our vigilance is not partisan." nsubj, poss, our, vigilance, partisan (vigilance, poss, our), (partisan, nsubj, vigilance) "we shall now hear mr wurtz speaking against this request." hear, aux, shall, speak, nsubj, wurtz, hear, ccomp, speak (hear, aux, shall), (speak, nsubj, wurtz), (hear, ccomp, speak) "this seems to me to be a workable solution." solution, amod, workable, seem, xcomp, solution (solution, amod, workable), (seem, xcomp, solution) ## discussion: handling the long tail the core of each natural language is a set of words (a lexicon), and a set of grammar rules that define how to combine the words into meaningful sequences. the lexicon is divided into content words, which possess semantic content, and function words, which denote the grammatical relationships between content words. the function words belong to closed classes, for example prepositions, that normally do not accept new words, while content words belong to open classes; new nouns, for example, are coined regularly. the long tail of the zipfian distribution of word frequencies consists of content words while the grammar-enforcing function words are mostly in the head of the distribution. recent studies suggest that the distinction between content and function words as well as the zipfian distribution are important for (compositional) generalisation to arise: steinert-threlkeld (2020) provides empirical evidence that function words enable robust forms of @xcite found that the zipfian distribution helps language models strike a balance between memorisation and (in context) generalisation. as shown also by @xcite , memorising some of the long tail is not in conflict with generalisation; on the contrary, it enables optimal generalisation. at the same time, some studies have shown that neural nlp system have some difficulties with handling the long tail, or at least handling the tail in a way that we as users of the models would expect and want. @xcite and @xcite found that nlp systems' performance is heavily influenced by word frequency in training. @xcite compared languagemodel-generated text to a reference and found that the models underestimate the long tail of wellformed sequences; furthermore, this probability mass didn't go to the head of the distribution but rather the models overestimate the probability of ill-formed sequences. as noted in previous sections, the desired distribution of atoms in the dbca framework contrasts with the zipfian distribution found in natural language. as keysers et al. (2020) explain, they design their cfq benchmark "so as to have few and meaningful atoms" which means there is no long tail of infrequent primitives. this is related to having a close-to-zero atom divergence: if the atom distribution had a long tail, it would not be easy to have the same relative atom distributions in training and test sets since at least those atoms that appear only once would make the distributions diverge. this harks back to the question of how compositional generalisation could be assessed in purely natural tasks, where every rule has an exception, and where idioms and irregularities muddle the systematicity (as discussed by @xcite ). our benchmark provides one answer to this question. although we use the principles of the dbca framework regarding distribution divergence, we don't make the corpus less natural by artificially shrinking the vocabulary size of the corpus. instead, to make the divergence calculations manageable in practice, we leave some of the vocabulary out of the calculations. a downside of this choice is that the test sentences in the maxd c splits don't contain exclusively novel dependencies. the advantage is that the vocabulary is similar to that in the original real-world natural language dataset, while the test set includes an increased number of novel dependencies to test generalisation. ## conclusion our genbench collaborative benchmarking task submission consists of train-test splits of the europarl parallel corpus with divergent distributions of dependency relations. these data splits can be used to assess the ability of nmt systems to translate novel dependency relations in a purely natural language translation task. we derive the data partitioning method from the distribution-based compositionality assessment framework, which provides generalisable principles of how to assess compositionality. our application of the dbca framework is straightforward to extend further to new datasets and languages, and to any other nlp task where the training and test sets consist of sentences, such as paraphrase detection. the dbca framework is useful for real-world natural language tasks too, even though it was originally designed for a more artificial data setting. it should be kept in mind, however, how the principles of the dbca framework diverge from the reality of natural language data.