Comparing the Evaluation and Production of Loophole Behavior in Humans and Large Language Models (2023)

Abstract: In law, lore, and everyday life, loopholes are commonplace. When people exploit a loophole, they understand the intended meaning or goal of another person, but choose to go with a different interpretation. Past and current AI research has shown that artificial intelligence engages in what seems superficially like the exploitation of loopholes, but this is likely anthropomorphization. It remains unclear to what extent current models, especially Large Language Models (LLMs), capture the pragmatic understanding required for engaging in loopholes. We examined the performance of LLMs on two metrics developed for studying loophole behavior in humans: evaluation (ratings of trouble, upset, and humor), and generation (coming up with new loopholes in a given context). We conducted a fine-grained comparison of state-of-the-art LLMs to humans, and find that while many of the models rate loophole behaviors as resulting in less trouble and upset than outright non-compliance (in line with adults), they struggle to recognize the humor in the creative exploitation of loopholes in the way that humans do. Furthermore, only two of the models, GPT 3 and 3.5, are capable of generating loopholes of their own, with GPT3.5 performing closest to the human baseline.

## introduction imagine a child poking at their beans, dreaming of dessert. their exasperated father tells them, "you can't have dessert until you eat some beans." the child groans, but then lights up, eats two beans, and holds out their hand for a cookie. the father rolls his eyes and begins saving up for law school. this commonplace example showcases the exploitation of loopholes: a person understands what is asked of them, but does not want to comply with the request, nor disobey it outright. in this grey area, they instead act on an unintended interpretation of the directive. the underlying mechanics of loophole behavior are quite sophisticated, and require an understanding of pretense, pragmatics, planning, and value. despite this cognitive complexity, everyday experience, as well as recent research, suggests that loophole-seeking is frequent, intuitive, and emerges in children as young as 5 years of age @xcite . loopholes have been a source of amusement and headache in fable and history dating back centuries. but more recently, the behavior of agents that 'do what you ask, but not what you want' has become a source of concern for people who study machine intelligence, as well as policy makers interested in ai safety @xcite . the problem is not restricted to a particular model or algorithm, and there are scores of examples of different kinds of systems gaming their task specifications to minimize a loss function, or achieve an objective in a way unintended by the people who specified it @xcite . such machines are described as 'creative' or 'cheating' or 'genie-like', but it should be stressed that they are not engaging in loopholes in the sense that they recover the original goal or intent and choose to act on a different interpretation. rather, such algorithms are exactly maximizing a given loss function or achieving a given goal. it is the human designer that realizes that the goal being achieved is not the one they intended. complaining that such systems are cheating is like saying a bridge that fell down is lazy because it didn't want to stay up. still, such failures are revealing of the human-side challenges of fully and accurately specifying one's goals and intentions, especially as models grow more complex and it becomes more difficult to evaluate their capabilities. despite the concern with loopholes in ai and machine learning, and the existence of many examples figure 1: task overview. we assess loophole behavior in models and humans using two tasks: evaluation of compliant, loophole, and non-compliant behaviors on the metrics of trouble, upset, and humor (left) and generation of loopholes (right). testing loophole behavior explicitly in llms is useful for at least three reasons: first, it helps us better understand the scope and limits of pragmatic reasoning abilities in llms. these models are taken by some researchers as models of human reasoning and language understanding, and a better understanding of the scope and limitations of llms in capturing loophole behavior can also help inform cognitive models of this behavior in humans (e.g., @xcite . given that there is an increased understanding that llms do well at formal linguistic competence, but not at pragmatic language use @xcite , then to the degree that llms succeed in such tasks, they can help isolate what aspects of loophole rea-1 given the pace of advances in llms, any more specific statement about their current state would likely be outdated by the time this paragraph is read. soning may be "solved" without further specialized reasoning about value, pretense, or mental states. if they don't, then hypotheses about how this reasoning is carried out in humans can help build out scaffolds and structures to support this reasoning in llms. second, as a phenomenon, loophole behavior subverts the usual cooperative assumptions that are at the heart of pragmatics @xcite : among humans, the loophole actor can pretend they were trying to be compliant by exploiting the ambiguity inherent in language and social interaction for their own ends (i.e., claiming they honestly misunderstood). so, this behavior provides an important test bed for potentially hostile machine abilities. third, explicitly testing loophole behavior as a task in machines helps address ai safety concerns that have relied on indirect examination. in this work, we compared the performance of several different llms to humans on two tasks designed to assess the understanding of loopholes 2 . by loopholes, we mean a class of behaviors where people intentionally misunderstand a given request, favoring a less likely though still possible interpretation in the service of one's own goals. we used two different tasks to assess loophole behavior: evaluation and generation. in the evaluation task, models and human participants were given vignettes that describe the actions of different protagonists (compliance, non-compliance, or loophole) when presented with a directive from another person. models and humans were asked to evaluate the protagonist's behavior in response to the directive on three metrics: how funny the behavior would be to the other person, how much trouble the protagonist will get in with the other person for their behavior, and how upset the other person would be about the behavior. the expectation is that loophole behavior will result in more trouble and upset than compliance, but less so than outright defiance. further, the creative exploitation of loopholes is expected to introduce an element of humor that neither compliant nor non-compliant behaviors would. in the generation task, models and humans were presented with vignettes that, in addition to the directive, describe the intentions of the protagonist (always at odds with the intention of the directive), and asked to provide a loophole that the protagonist could exploit. we assessed models of various sizes and training objectives on these tasks: tk-instruct @xcite , flan-t5 @xcite , @xcite , instructgpt @xcite , and gpt-3.5 (chatgpt). on the evaluation task, we find that while many of these models succeed in differentiating the relative amounts of trouble a protagonist would get in for compliant, non-compliant, and loophole behaviors, fewer models are able to effectively reason about the upset that another person would experience as a result of these behavior types. further, none of the models we test differentiate these behavior types on how funny they would be to another person, suggesting an inability to perform the more complex reasoning about social conventions and expectations that allows people to recognize the humor in the creative exploitation of loopholes. when it comes to loophole production, we find that gpt-3.5 is the only model that approaches the human baseline, with loopholes far outnumbering all other response categories. gpt-3 comes close, producing only slightly more loopholes than non-compliant actions, while the remaining models largely produce actions that range from non-compliance, to negotiation, to lying, but most often, incoherent or irrelevant responses. ## experimental paradigm we evaluate humans and models on two behavioral tasks: evaluation of the costs and rewards of engaging in compliant, non-compliant, or loophole behaviors, and generation of loopholes. evaluation stimuli are taken from @xcite , consisting of 36 scenarios that describe a directive for the protagonist. for the evaluation task, social consequences of the protagonist's behaviors were measured through three metrics: (1) how much trouble the protagonist would get into performing the behavior, (2) how upset the other person would be about the protagonist's behavior, and (3) how funny the other person would find the protagonist's behavior. these metrics reflect critical distinctions among compliant, loophole, and non-compliant behaviors, and children as young as five years are shown to differentiate loophole behaviors when reasoning about these consequences @xcite . in line with @xcite , we posit that ratings of trouble and upset, as metrics of penalty, are essential to the generation of loopholes, as the costs associated with direct refusal to a speaker's request may trigger search about possible alternative actions that the agent could take to achieve their own goals. rating of humor, on the other hand, can be thought of as a reward for an action that is recognized as clever or unexpected, and may help counteract the penalties of ultimately not complying. taken together, these metrics help assess the social vs. egocentric components of loophole behavior: while trouble describes the direct consequences of one's own behavior, both humor and upset require higher-order reasoning about others' mental states. in the generation task, we study the ability to generate reasonable actions that fall in the grey area between full compliance and outright noncompliance. this rounds out our assessment of loophole behavior because once the costs of noncompliance have been assessed and the decision to engage in loophole behavior has been made, one must be able to reliably calibrate the grey area between personally displeasing compliance and costly non-compliance and produce the alternative actions that fall within this space. ## discussion the metrics that we use to assess models' ability to evaluate different behavior types-trouble, upset, and funny-can be thought of as representing two different aspects of costs and rewards. a more egocentric perspective is assessed by the trouble metric, where the models have to reason mostly about the direct consequences of an action. however, reasoning about the amount of upset those actions might cause to another person or how funny they might find such actions, requires higher-order, theory of mind reasoning. further, among the latter two metrics, humor is thought to require especially complex reasoning about social conventions and expectations, with some theories suggesting that humor arises from a violation of our mental patterns and expectations @xcite . we find that the models' performance track on the eval-uation task with the graded social and pragmatic complexity of these metrics. while most models are able to reason about the direct consequences of a protagonist's actions through the amount of trouble they would get into, fewer are able to perform the higher-order reasoning needed to differentiate these behaviors on upset, while none managed to do so for humor. these findings corroborate with those of recent work that have studied humor in llms @xcite , suggesting that models continue to struggle to reason about complex social conventions and expectations. while the phenomena we study have been found to be sensitive to a variety of cultural and demographic factors @xcite and would benefit from further exploration with more culturally diverse populations, the models we test are known to have been trained on text data that reflects a similar distribution of cultural backgrounds as the population that participated in our behavioral studies. thus, the models' failures may reflect a lack of a deeper cognitive understanding of the expectations, social norms, and incongruities needed to reason about the social consequences of loopholes and echo the growing skepticism about recent models' theory of mind abilities (e.g. @xcite @xcite . we also note some effects of model size and training. at 250m parameters, flant5-base is significantly smaller than any other model we tested, and also performs the worst across both the evaluation and generation tasks. in addition to their smaller parameter size compared to the gpt models, all the t5-based models we tested also have smaller data input than the gpt models, with t5's c4 corpus comprising just 60% of gpt-3's training data. when comparing the performance of our two families of t5-based models, tk-instruct's fine-tuning on the larger and more diverse sup-natinst benchmark (compared to the flant5 task set) appears to translate to improvements on our task. still, there appear to be interesting exceptions to the benefits of scaling on our task. for example, instructgpt shows comparable performance to the two tk-instruct models on the generation task, despite being significantly larger than either (175b parameters vs. 3b and 11b). this could be because instructgpt was specifically fine-tuned to improve alignment and better follow instructions ( https://openai. com/research/instruction-following ), suggesting that instruction fine-tuning may not necessarily improve the flexibility of pragmatic reasoning. our study adds to a growing body of work evaluating llms' understanding of various pragmatic phenomena @xcite @xcite @xcite . some of these phenomena, like deceit @xcite , approach the spirit of loopholes by probing understanding of misaligned values. similar to the evaluation of conversational implicature in @xcite , our evaluation task also probes understanding of intentions, while additionally testing the costs and values associated with agreeing or refusing to comply with them, analogous to recent social commonsense reasoning benchmarks @xcite . our generation task goes beyond any of these formats to probe models' ability to produce pragmatic behavior, as opposed to choosing between answer options. when given information detailing the misalignment between the goals of different agents (e.g. the other person wants x, the protagonist wants y), we assess whether models are able to generate reasonable actions that fall in the grey area between full compliance and outright noncompliance. ## conclusion in this work, we compared the performance of several different llms to humans on two tasks designed to assess loophole comprehension. in the evaluation task, we find that a number of models, of varying sizes, are able to differentiate compliant, non-compliant, and loophole behaviors based on how much trouble the protagonist would get in for their action. however, fewer models are able to effectively differentiate these behavior types when it comes to reasoning about others' emotional reactions in response to such behaviors. we find that only three of the models, gpt-3.5, gpt-3, and tk-instruct-3b reflect human baselines on the metric of how upset the other person would be about the protagonist's behavior, in that they significantly differentiated loopholes from non-compliance and rated compliance with low levels of upset, with gpt-3.5 performing best . for the final metrichow funny the protagonist's behavior would be to the other person-no model was able to recognize the humor in the creative exploitation of loopholes in the way that humans do. when it comes to loophole generation, we find that only gpt-3.5 approaches the human baseline, with loopholes far outnumbering all other response categories. gpt-3 comes close, with slightly more loopholes than non-compliant actions. in the future, we are interested in how loophole behavior can be used as a testbed for developing techniques that could equip smaller models with similar human-like pragmatic reasoning capacity.