Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses (2023)

Abstract: At the staggering pace with which the capabilities of large language models (LLMs) are increasing, creating future-proof evaluation sets to assess their understanding becomes more and more challenging. In this paper, we propose a novel paradigm for evaluating LLMs which leverages the idea that correct world understanding should be consistent across different (Fregean) senses of the same meaning. Accordingly, we measure understanding not in terms of correctness but by evaluating consistency across multiple senses that are generated by the model itself. We showcase our approach by instantiating a test where the different senses are different languages, hence using multilingual self-consistency as a litmus test for the model’s understanding and simultaneously addressing the important topic of multilingualism. Taking one of the latest versions of ChatGPT as our object of study, we evaluate multilingual consistency for two different tasks across three different languages. We show that its multilingual consistency is still lacking, and that its task and world understanding are thus not language-independent. As our approach does not require any static evaluation corpora in languages other than English, it can easily and cheaply be extended to different languages and tasks and could become an integral part of future benchmarking efforts.

## introduction the staggering pace at which the capabilities of large language models (llms) have increased in the recent past comes with many questions related to what kind of progress we are making on the road towards true machine intelligence and human-level understanding. to assess such progress, practitioners often rely on benchmarks that measure natural language understanding (e.g. @xcite , commonsense reasoning (e.g. @xcite , or probe for factual knowledge (e.g. @xcite , among other things. the extent to which such benchmarks can be used to assess whether llms "understand" language is widely debated (e.g. @xcite . often mentioned concerns in this context are that llms may learn specific lexical patterns rather than general principles (e.g. ray @xcite and, relatedly, that benchmark scores may confuse competence in form with competence in meaning (e.g. @xcite . in support of these concerns, llms have been found to bypass certain tasks by relying on memorised information from the training data @xcite . more recently, the enormous amount of data that models are trained on and the fact that this data is often not publically accessible have further increased the difficulty of assessing whether benchmarks really quantify what they are meant to quantify. a benchmark always makes assumptions about what a model has seen in its training phase, and, given the rapid changes on that front, it is difficult to design challenging benchmarks that remain informative past training rounds of new models. in addition, novel evaluation data may leak into the training data of newly trained models foot_0 -which even the most future-proofed benchmarks may not withstand. in this paper, we propose a novel approach to evaluate models' task or world understanding that aims to create some separation between form and meaning in benchmarking and simultaneously mitigates the challenging evaluation-contamination loop. our method is based on the idea that language is used to describe or act in the world @xcite and that this world functions as an anchor for diverse linguistic forms. having a genuine understanding of the world thus implies consistency among different linguistic expressions that pertain to the same entities within the world. to give an figure 1 : illustration of the basic mechanism of our paradigm: we use the model to generate other senses of the original input. the model's answers on the original input and the alternative sense are used to evaluate its consistency. in this example, the model is presented with the task of paraphrase detection in english (sentences taken from paws-x) and generates another sense by translating from english to german. example, if you ask a colleague who is fluent in both french and english if a particular statement is true, you expect their answer to be invariant to the language (french or english) in which you ask this question. we leverage this intuition to investigate whether models have a consistent world model across different senses (in the case above: languages) and, consequently, a consistent understanding of the tasks that they are asked to execute. loosely inspired by @xcite , we take different senses to be different modes of presentation or notational variants. crucially, rather than generating different senses ourselves, we ask the model to create different versions of the same question. this ensures that potential inconsistencies are really due to model-internal inconsistencies rather than misinterpretations of the question. additionally, the method is protected from data contamination: as the different senses are regenerated for every evaluation, they cannot leak into new training data. lastly, it can cheaply and easily be applied to already available benchmarks and therefore reduces the burden on data generation. our approach can be applied to a number of different senses. here, we showcase it focusing on the multilingual case described in the example above, by asking whether models are consistent across different languages (see figure 1 ). in essence, we are thus using multilingual self-consistency as a litmus test for their understanding, simultaneously addressing the important topic of multilinguality. taking one of the latest sota versions of chat-gpt foot_2 as our object of study, we evaluate multilingual understanding for two different tasks (paraphrase detection and natural language inference) across three different languages (chinese, german and english). it turns out that the model is inconsistent across all languages and tasks, despite being able to perform the tasks reasonably well in english and generating high-quality translations. taken together, our analyses provide strong evidence that the model's task understanding is modulated by the representational form of the task. in sum, we make the following contributions: with our work, we hope to not only present an interesting set of empirical results on multilingual consistency but also propose a novel, easily applicable method to generate many more challenging evaluation tests. our framework targets models that can follow instructions to generate alternative senses and are able to generate these senses based on these instructions. thus, with the growing popularity and capabilities of chat-models and instruction-tuned models, such as gpt-4 (openai, 2023) or llama-2 @xcite , our framework is becoming increasingly relevant. 3 2 related work existing benchmarks for evaluating language understanding in llms form the foundation for our work. the main idea of our paradigm is to evaluate llms in terms of their consistency across different senses of these benchmarks and is therefore related to other work on self-consistency in llms. in creating multiple senses through translation, there is also a close connection between our execution of this paradigm and multilingual evaluation. appendix a provides a genbench eval card @xcite ) that classifies our work in the context of generalisation research. evaluating language understanding. a wide range of benchmark tasks has been developed to evaluate specific aspects of natural language understanding in llms. to evaluate general language understanding across diverse tasks, genres, and datasets, several of these tasks have been combined into multi-task benchmarks, such as glue @xcite or superglue @xcite . these benchmarks focus on english and evaluate, among others, paraphrase identification (e.g. paws; zhang et al., 2019), natural language inference (e.g. mnli; @xcite , and commonsense reasoning (e.g. copa; @xcite . in response to the rapid improvements of llms on these benchmarks other multitask benchmarks have been developed. mmlu, for example, assesses world knowledge and problemsolving ability across a large number of subjects, covering stem, humanities, social sciences, and more @xcite . while our paradigm also makes an effort to find more appropriate evaluation methods, it not only assesses performance but also evaluates the model's ability to consistently solve a task across multiple languages, thereby providing insights into its ability to abstract from specific representational forms. ## methods we now proceed with describing the model ( § 3.1) and the benchmark data ( § 3.2) we use for our experiments, as well as the procedure we use for extracting translations from the model ( § 3.3). ## results in this section, we discuss the results of our experiments (see figure 2 ), beginning with our primary experiment in which we assess how consistent the model's task understanding is across languages ( § 4.1). in subsequent experiments, we investi-gate the individual effects of translating the dataset or the instructions ( § 4.2), and ensure that inconsistencies do not arise from inaccurate translations ( § 4.3) or poor task performance ( § 4.4). ## conclusion in this paper, we presented a novel paradigm for evaluating language models, which leverages consistency across different linguistic senses. our method can be used to assess generalisation ability beyond specific forms. it offers affordability and applicability to different evaluation tasks, while also mitigating the risk of evaluating on data that the model has already encountered during training. as such, multisense evaluation is not an alternative to current benchmarks but a complement. reporting consistency next to standard evaluation metrics will make model evaluation more meaningful in providing an estimate of how well the model understands a given task beyond its specific form. therefore, we encourage other researchers to treat multisense consistency as an essential part of benchmarking. to showcase the effectiveness of our paradigm, we conducted a multilingual multisense consistency evaluation of chatgpt (gpt-3.5-turbo), a sota llm. the results of this evaluation unveiled significant inconsistencies across different senses generated by the model itself through translation, suggesting a lack of genuine, cross-sense task understanding. to ensure the validity of this interpretation, we ruled out alternative explanations such as model-subjective or objective changes in meaning caused by the translation as well as inadequate performance on the original task. collectively, these findings show that chatgpt exhibits a languageand therefore sense-dependent task understanding, which might also affect other leading llms. our paradigm can be cheaply and easily expanded to include more languages, tasks, models, and notions of "sense". our choice to generate multiple senses through translation is wellsuited for evaluating current and future models, given the growing trend towards multilingual mod-els with increasingly proficient translation abilities. nevertheless, numerous other multisense evaluations are conceivable. for instance, instead of using model-internal translations, one could employ model-internal paraphrases. multiple senses could also be generated in different domains, such as arithmetic (different formulas yielding the same result) or code (different functions producing the same input-output mapping). last but not least, calculating consistency for various tasks may help disentangle "unfounded" language-specific differences (forming the focus of our analysis) from differences related to cultural bias. in conclusion, multisense consistency can be applied as long as the model under investigation can create different senses of a given task and has some understanding of the task in its original sense. it offers the possibility of evaluating the task understanding detached from a specific task realisation, and we hope it will contribute to making standard benchmark evaluations more meaningful.