m SCAN : A Dataset for Multilingual Compositional Generalisation Evaluation (2023)

Abstract: Language models achieve remarkable results on a variety of tasks, yet still struggle on compositional generalisation benchmarks. The majority of these benchmarks evaluate performance in English only, leaving us with the question of whether these results generalise to other languages. As an initial step to answering this question, we introduce mSCAN, a multilingual adaptation of the SCAN dataset. It was produced by a rule-based translation, developed in cooperation with native speakers. We then showcase this novel dataset on some in-context learning experiments, and GPT3.5 and the multilingual large language model BLOOM as well as gpt3.5-turbo.

## introduction humans learn quickly by easily recombining previously known concepts in unseen settings. several benchmarks have been designed to empirically investigate whether neural networks are equipped with similar abilities @xcite @xcite . such benchmarks are composed of tasks in which the training data and the test data have different and carefully chosen distributions. recent work used these benchmarks to evaluate pretrained large language models (llms) and showed that despite their remarkable success on many other tasks they still struggle with compositional generalisation @xcite . the majority of the research on compositional generalisation has focussed on english data and models -but do compositional generalisation abilities differ across languages? indeed, it has been argued that the performance of a model in english is not a guarantee that it will work "equally or even reasonably well" in other languages @xcite . on top of that, compositional generalisation itself is not guaranteed to work uniformly across human languages @xcite . furthermore, the exploration of cross-and multilingual compositional generalisation could benefit the expansion of language technology to lowresource languages and settings @xcite , as a potential approach to overcome the need for huge amounts of data that neural models require. with ever-increased scale, some large language models have shown great performance on downstream tasks while only conditioned on a few examples, and without updating their parameters. this is known as in-context learning, a paradigm in which some very large models such as gpt-3 and palm have been shown to manifest reasoning abilities when prompted in specific ways, including in multilingual settings @xcite . despite these promising perspectives, it does not currently stand as an alternative to fine-tuning. some recent research has sought to investigate compositional generalisation within the in-context learning paradigm, showing it gets outperformed by smaller fine-tuned models. as a means to further the study of compositional generalisation in multiple languages, we introduce mscan (multilingual scan), an adaptation of the scan benchmark into french, hindi, mandarin chinese and russian. we also provide for each language both the original scan benchmark splits (add_jump, add_turn_left, length) as well as the maximum compound divergence splits @xcite . we also present preliminary experimental results using mscan in an in-context learning paradigm on bloom and gpt3.5-turbo. following the genbench taxonomy @xcite , the primary motivation for this work can be characterised as intrinsic given its primary function to provide a means to evaluate compositional generalisation in multilingual settings. similarly to the original scan benchmark, the source of the distribution shift is fully generated and its motivation practical cognitive intrinsic fairness □ generalisation type compositional structural cross task cross language cross domain robustness □ shift type covariate label full assumed □ shift source naturally occuring partitioned natural generated shift fully generated □ shift locus train-test finetune train-test pretrain-train pretrain-test □ figure 1: genbench evaluation card type is covariate. moreover, the in-context set-up of our experiments places the shift locus between the pre-train and test stages though we note that the data can also be used in a fine-tuning setup in the future. ## background pre-trained multilingual models seek to address the challenge of low-resource languages, by leveraging the pre-training and the hope that high-resource languages will help lower-resource ones. largescale multilingual language models have achieved impressive performance across typologically distinct languages @xcite . yet, the cross and within-language performance of downstream tasks on such models remain correlated to their amount of language-specific pertaining data @xcite . however, if scaling up the amount of pretraining data might improve cross-lingual generalisation, it might come at a price when it comes to compositional generalisation. @xcite have questioned the reported benefits of pre-training on compositional generalisation benchmarks and have observed a case of inverse scaling, where the performance degradation on cogs actually increases with the amount of pre-training data. in a further study on the impact of model scale on compositional generalisation, qiu et al. (2022) compared fine-tuning, prompt-tuning and in-context learning on multiple compositional generalisation datasets and observed that for in-context learning, the performance is correlated with model size. however, it is worse than for fine-tuned, smaller models. datasets they used included cogs and the compositional freebase question dataset or cfq @xcite , which consists of questions and answers in natural language, as well as accompanying sparql queries against a knowledge base. @xcite evaluated four model families for in-context learning on multiple semantic parsing benchmarks. despite their observation that the larger models tend to do better, they report that the in-context learning performance on scan and cfq is very small for the models tested. mcwq @xcite , a multilingual variant of cfq, is the first adaptation into multiple languages of a compositional generalisation benchmark. it was created with the use of neural machine translation. wang and hershocovich (2023) have shown that using neural machine translation to translate already existing benchmarks entails "critical semantic distortion", and favour a rule-based translation of the mcwq dataset. the msgm benchmark @xcite investigates the mathematical reasoning abilities of llms in multilingual settings, by providing data in ten different languages. even though the decomposition of scan commands closely resembles that of arithmetic operations, the msgm differs in that it does not specifically target the capacity of the model to map forms to a representation of meaning. as such, there has not yet been any investigation specifically targeting the compositional generalisation abilities of multilingual models in an in-context setting. ## the mscan dataset our goal was to adapt the simplified version of the commai navigation dataset or scan @xcite to languages that belong to typologically diverse families and typically are represented in varying proportions in the training data of multilingual models. the languages selected also have different language scripts: latin, cyrillic and devanagari. the original scan consists of a set of navigation commands in english such as "jump left", and their corresponding sequence of actions, such as lturn jump. it is a synthetic dataset: the natural language commands are generated by a phrase-structure grammar, and the actions are produced by applying a semantic interpretation function. as such, it is akin to a semantic parsing task. ## conclusion the majority of the research on compositional generalisation is focussed on english, leaving open the question as to whether its findings can generalise across languages. as an initial step towards this exploration, we introduce mscan, a multilingual adaptation of the scan dataset, produced using rule-based translation, with rules developed in cooperation with native speakers. we then showcase this novel dataset on some in-context learning experiments, with the multilingual large language model bloom.