I am a Strange Dataset: Metalinguistic Tests for Language Models (2024)

Abstract: Statements involving metalinguistic self-reference (“This paper has six sections.”) are prevalent in many domains. Can large language models (LLMs) handle such language? In this paper, we present “I am a Strange Dataset”, a new dataset for addressing this question. There are two subtasks: generation and verification. In generation, models continue statements like “The penultimate word in this sentence is” (where a correct continuation is “is”). In verification, models judge the truth of statements like “The penultimate word in this sentence is sentence.” (false). We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all. The dataset is hand-crafted by experts and validated by non-expert annotators. We test a variety of open-source LLMs (7B to 70B parameters) as well as closed-source LLMs through APIs. All models perform close to chance across both subtasks and even on the non-self-referential metalinguistic control data, though we find some steady improvement with model scale. GPT 4 is the only model to consistently do significantly better than chance, and it is still only in the 60% range, while our untrained human annotators score well in the 89-93% range. The dataset and evaluation toolkit are available at https://github.com/TristanThrush/i-am-a-strange-dataset

## introduction self-reference plays a crucial role in the way we think about mathematics @xcite , theoretical computer science @xcite , recursive programming @xcite , understanding complex cases in hate speech detection @xcite , aptitude tests @xcite , and comedy @xcite . some positions in the philosophy of mind consider self-referential capabilities to be a key aspect of higher intelligence if someone asks whether this sentence has a capital letter, the correct answer is   figure 1 : an example highlighting the challenge presented by our task. all models that we tested on our dataset are close to chance-level. or even consciousness @xcite . of course, self-reference is also pervasive in how we communicate: at least one paper you read today is bound to contain "in this paper" @xcite . in this paper, we focus on metalinguistic selfreference, the complex kind of self-reference in which language is used to make claims about itself, as in "this sentence has five words" and "this paper has six sections". 1 using such language involves reasoning about metalinguistic properties (counting words, naming parts of speech, etc.) and resolving self-reference. humans generally have no trouble with such language, and may even enjoy its playful and sometimes paradoxical nature @xcite @xcite . recently, large language models (llms) have demonstrated striking cognitive capabilities @xcite @xcite @xcite @xcite . but do they have the same mastery over metalinguistic self-reference as we do? see figure 1 for an example of the issue that llms face. to help address this question, we present a new task and dataset called "i am a strange dataset". we are inspired by douglas hofstadter's explorations of self-reference in language @xcite @xcite , and borrow part of the name from one of his books: "i am a strange loop" @xcite . an example in "i am a strange dataset" is comprised of two self-referential statements that begin in the same way but have different endings (figure 2 ). one is true and one is false. crucially, the ending flips the truth value of the overall statement. there are two subtasks: generation and verification. in generation, the model must generate the true statement and reject the false one. in verification, models judge the truth of completed statements. to complement the main self-referential data, the dataset also contains metalinguistic nonself-reference examples. these are minimally different from the main examples and serve as controls to assess whether models can reliably handle metalinguistic statements in the absence of selfreference. in addition, all the examples in the dataset are tagged by expert annotators to further aid in error analysis. the first and last words of this sentence are "the" and "respectively", respectively. the first and last words of this sentence are "the" and "the", respectively. figure 2 : examples from the dataset. each example is comprised of a beginning and two different endings. one of the endings makes the statement true, but it would make the statement false if it referred only to the beginning. the other ending makes the statement false, but it would make the statement true if it referred only to the beginning. true endings are on the left and shown in blue. false endings are on the right and shown in red. in the case of the code example, the true continuation is shown above the false one. the knowledge required to solve the problems need not be. "i am a strange dataset" follows a simple schema that requires self-referential language, and consequently tests an array of metalinguistic capabilities. as far as we know, this is the first ai challenge dataset targeting metalinguistic selfreference, although there are metalinguistic evaluations @xcite and corpuses @xcite . ## i am a strange dataset in this section, we describe how the dataset is constructed and how we measure model performance. ## results results in this paper for claude 2 (anthropic, 2023), gpt 3.5 turbo (brown et al., 2020), and gpt 4 (openai, 2023) were collected through the gpt-4, gpt-3.5-turbo, and claude-2 endpoints on their respective apis on jan 7, 2024 (unless otherwise stated). note that the claude 2 api does not support logprobs, so the only metric that we report for it is the text-based cot validation metric. the openai api supports top-5 logprobs access, and it turns out that this is enough to get logprobs for "true" vs "false" in our experiments. it is not enough to get the multi-token logprobs required for the generation metric, though. the cot metric gives us extra insight into the limitations of models because we can see if their reasoning aligns with their final answer. in many cases, they make fairly obvious mistakes as seen in figure 3 . in some cases, the models choose the correct answer even though their reasoning is flawed, as seen in figure 5 . it is important to note that there are limitations with the cot validation metric, and the zs and fs non-relative validation metrics too. these metrics are about judging a statement as true or false, independent of the statement with the alternative continuation. the dataset schema calls for sentences which are either true or false, but in practice, language can be ambiguous. figure 6 illustrates. a fairer gauge of model capability may be to compare the continuations with each other, as the generation and relative validation metrics do. tags. we present the average logprobs-based model scores for the tags in non-self-referential control. we provide the difference in scores between the non-self-referent examples and analogous original examples in gpt 4: the statement is asking why it doesn't have an answer, but it does have an answer. so, the statement is false. figure 6 : this is a particularly ambiguous example that is assigned the gold label of true. the response from gpt 4 is marked as incorrect, although one could make a case that the statement is false because of the false presupposition which is corrected later in the answer. the generation and relative validation metrics are robust to these situations. for these metrics, the losses from "it does though." versus "it just doesn't." (the alternative continuation) are compared directly. regardless of ambiguity about whether a statement is true outright, we can notice a higher confidence for "it just doesn't." as the false continuation. we find that our dataset is so hard due to nonself-referential metalinguistic aspects that models score around chance here too, with the exception of gpt 4. it is the only model which is strong enough to perform significantly above chance for every metric tested on the main dataset in ## conclusion a grasp of self-reference is important in a variety of domains, and is a notable aspect of human intelligence. we introduced a novel task and dataset to assess the capacity of models to generate and score across all of the logprobs-based metrics. we only evaluate five 7b models, and two models for each of the other sizes, so computing confidence intervals for each point is not particularly informative. under the null hypothesis that parameter size has no effect on score, we can compute the p-value for these results as a whole nonparametrically: there are 24 ways that these 4 points can be arranged and in only 1 of the ways do they all increase with the parameter count: @xmath0.042. our results indicate that this dataset is hard not only due to the self-referent part of a self-referential statement. the challenge also comes from other metalinguistic aspects, such as recursively applying arithmetic operations on sentences. still, there is some limited evidence that gpt 4 struggles more with self-referential metalinguistic problems than analogous non-self-referential problems.