clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents (2023)

Abstract: Recent work has proposed a methodology for the systematic evaluation of “Situated Language Understanding Agents” — agents that operate in rich linguistic and non-linguistic contexts — through testing them in carefully constructed interactive settings. Other recent work has argued that Large Language Models (LLMs), if suitably set up, can be understood as (simulators of) such agents. A connection suggests itself, which this paper explores: Can LLMs be evaluated meaningfully by exposing them to constrained game-like settings that are built to challenge specific capabilities? As a proof of concept, this paper investigates five interaction settings, showing that current chat-optimised LLMs are, to an extent, capable of following game-play instructions. Both this capability and the quality of the game play, measured by how well the objectives of the different games are met, follows the development cycle, with newer models generally performing better. The metrics even for the comparatively simple example games are far from being saturated, suggesting that the proposed instrument will remain to have diagnostic value.

## introduction there is an old joke: a guy has a dog that plays checkers. "my goodness," everyone says, "that's amazing. what a brilliant dog!" -"not really," he replies, "i beat him four games out of five." this joke nicely reflects where we are with * contributions: kc coded and analysed the wordle games. jg did so for taboo, and managed the code repository. sh coded and analysed drawing and reference, implemented the llm interfaces, and edited the appendix. bm coded and analysed private/shared, and organised the evaluation computations. ps coded taboo and implemented the main framework. ds conceived the original idea, managed the project, and edited the main part of the paper. ## results as detailed in the appendix, the full benchmark (v1.0) consists of 250 instances: 30 for taboo, 30 for wordle, 30 for wordle+clue, 30 for wordle+clue+critic, 40 for drawing, 40 for reference, and 50 for private/shared. when it comes to the quality of the game play (in those episodes played to completion), we see a similar trend, with gpt4 overall performing best. we also see that there is ample room for improvement, with the best average score standing at 60.59. an outlier in terms of quality is wordle, where even though almost all models manage to stick to the local rules (produce a 5-letter word), even the best-performing model, gpt4, only reaches 4.56 on the quality metric, indicating that very few games are actually solved, and those only at the last attempt. this indicates that all models fail at integrating the feedback across turns and using it to constrain their guesses. the capabilities of dealing with verbal meaning definitions are shown by the large improvement that wordle+clue exhibits (to 47.89). interestingly, gpt4 is able to profit from (via another instance, self-)criticism, improving further to 50.11. again somewhat surprisingly, performance on the "multimodal" games (which require verbalisation of character-based graphics) is not bad. for drawing, as a deviation from the trend, gpt3.5 proved to be better at sticking to the game format (97.5% of episodes played to completion), although gpt4 still reached higher quality on its completed games. reference sees claude performing best, against the trend for all other games. ## discussion insights from taboo game: the results show that the open source models and luminous cannot, or only badly, play the taboo game. claude shows a strong performance with 76.92% played games and a quality score of 68.75%. the best scores are achieved by the gpt-4/4 pair with 94.92% played games and a quality score of 76.19%. we hypothesise that this is an effect of rlhf training time so that the model is highly aligned with the prompts given by the user (the game master). an indication is given by the increase from 28.81% to 69.49 % of games played when comparing gpt-3 (a foundation model) and gpt-3.5 (an rlhf fine-tuned model) where the quality score remains similar. both models share the knowledge representation and are similarly good in retrieving that knowledge, but the latter is better aligned. insights from reference and drawing games: claude and gpt 3.5 and 4 models get the best played ratio, which indicates that the generated outputs match the expected format. as this game is single turn, unlike the other games, errors cannot accummulate over turns. in drawing, luminous, claude and the open access models did not manage to follow instructions. the generated outputs included the repetition of the text in given instructions, which leads for games to be aborted. the played ratio of gpt 3.5 is higher than gpt-4. by looking at some selected instances, we saw that the outputs from gpt-4 are simply the appended text of multiple turns (on player a side) instead of generating each instruction separately in a single turn. gpt-3.5 is better at following the instructions (97.5 vs. 77.5 in played score) but gpt-4 is better at getting the target drawing (60.2 vs 89.0 in quality score), in those cases where the format was correct. insights from the scorekeeping game: games were aborted mostly due to the models failing to use the correct player tag. this is particularly complicated in this game because we are trying to simulate a multi-party conversation, with one tag for each interlocutor. interestingly, sometimes a mere reprompt with a generic addition (e.g. "please answer this question carefully.") would trigger it to generate the right tag, even though the mistake was not added to the history. another issue is that sometimes models would anticipate or invent slots and upcoming turns. anticipating is not totally incorrect, but it makes it harder for the gm to check for the private/shared status of a slot. claude and gpt-4 played the slot filling part very well; their mistakes came mostly from the scorekeeping component, with mixed results in abstract and concrete domains. in almost all cases, their main type of mistake was considering shared slot values to be still private. insights from the wordle game: models other than gpt-4 could not adhere to the game rules in at least half of the episodes they played. a significant observation is that most of these models did not incorporate the letter feedback to enhance their subsequent word guesses. this is evident from the repetition of letters from previous guesses in subsequent iterations. figure 19a illustrates this observation. the turn at which a correct guess is made provide insights into the efficiency of the guessing strategy. in the traditional wordle variant, gpt-4 takes an average of four turns (refer to ## related work playing games and learning from self-play stands at the beginnings of the "deep learning revolution" @xcite . 5 what is different here is the zero-or few-shot nature of our test, where the testing mode is different from the learning mode-this of course only being enabled by "foundation models" @xcite . the latest-apparent-qualitative jump has only recently been taken, so there are not that many papers yet that attempt a systematic evaluation; see, inter alia, @xcite . to our knowledge, game play of the kind proposed here has not yet been used for the systematic evaluation of these models. the idea of testing game play is mentioned in @xcite , and also already technically possible in @xcite , but has not been systematically executed there. a superficial similarity also exists to approaches like @xcite in that these approaches pair llms with scaffolding (as in our game master). a crucial difference, however, is that for us the task of the game master is to constrain the llm and to "keep it focused", as it were, on the game, rather than to extend its capabilities. park et al. (2023) also acted on the realisation that cllms can simulate agents which can be put into "self-play", but developed this idea in a different direction, towards investigating the emerging "social behaviour". newly proposed benchmarks such as @xcite , chatbot arena (lmsys, 2023) and open llm leaderboard (huggingface, 2023) focus on comparing models outputs either running them on existing datasets, employ human annotators to choose which output is preferred, or simply ask another llm to evaluate the outputs; these benchmarks do not test the interactive dialogue aspects of chat-based llms. another important aspect to note here is that using existing datasets for benchmarking might jeopardise the point of keeping the test instances unseen because those instances could have been part of the training data for these large language models. the datasets for clembench have been created from scratch and adding new games or new instances to the existing games is easy to ensure continued fair benchmarking. ## roadmap important next steps on our roadmap include testing the models' abilities to handle languages other than english and integrating the framework with the slurk chat tool @xcite in order to enable game play with human players. we also plan to experiment with games that have more than two players as well as games that require multimodal context such as images. we are also excited about the potential to use this as an instrument for testing models across size variations and training checkpoints, to analyse what it takes to acquire the capabilities tested here. lastly, with the measuring instrument introduced here in place, we can also turn to improving individual models (rather than testing existing models out of the box) so as to optimise their performance on a particular game or set of games. ## conclusions we have shown that current chat-optimised large language models can indeed serve as models of interactive agents, at least for controlled and ruleconstituted activities such as verbal games. we have described our general implementation of a framework for implementing rules to be played in "self-play" by such models, with the main idea being that a programmatic component, the "game master" can control the interaction and ensure that only formally correct moves are registered. we have described our example implementations and instantiations of such games, arguing that they span the breadth of the sub-capabilities involved in situated language processing (if only on a relatively superficial level). finally, we have shown that the evaluation of the game play can serve as an instrument to distinguish between models in terms of their language capabilities. with this work, we have aimed to open a complementary avenue for evaluating these models, beyond more classical reference-based nlp task evaluation or preferencebased evaluation, and into the realm of interactive language use. much remains to be done, but we hope that our framework can support some of this future work.