Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Models (2025)

Abstract: This study utilizes the game Codenames as a benchmarking tool to evaluate large language models (LLMs) with respect to specific linguistic and cognitive skills. LLMs play each side of the game, where one side generates a clue word covering several target words and the other guesses those target words. We designed various experiments by controlling the choice of words (abstract vs. concrete words, ambiguous vs. monosemic) or the opponent (programmed to be faster or slower in revealing words). Recent commercial and open-weight models were compared side-by-side to find out factors affecting their performance. The evaluation reveals details about their strategies, challenging cases, and limitations of LLMs.

## introduction the astounding abilities of large language models (llms) have led to what could be called a 'crisis of evaluation', where the previous paradigm of evaluating natural language processing (nlp) models-through pairs of problem instance and expected response-does not fit well any more. first, the main mode of usage of llms is through their embedding in a "chatbot", often across multiple turns, which is not represented by the referencebased evaluation mode. second, the closed nature and sheer size of the training data, often acquired through automatic means from the open internet, raises fears that the usual test datasets have been ingested and hence the training data has become contaminated, rendering the value of the tests even more doubtful @xcite . the use of games as an interactive environment where llms are tasked to perform certain actions and scored whether they achieve the task or not has emerged as a response to this situation @xcite @xcite , spymaster field operative gamemaster board 1. prompts for clue 2. prompts for guess mock opponent 3. & 5. updates game state 4. gets turn 2024), allowing the evaluation to more closely approximate the interactive use situation, and overcoming the data contamination problem in two ways. first, even for already documented games, it is easy to create new instances that lead to game play that differs from what is in existing data @xcite . second, to further extend the range of evaluated phenomena, it is only necessary to add new game implementations, rather than to create new datasets. it is this second dimension that we explore in this paper. we have implemented the game codenames foot_0 as a challenging means (already investigated in psycholinguistic studies, see below) for evaluating certain language-use capabilities. in this game, a first player needs to provide to a second player a clue which singles out certain words within a larger set of words given to both players (see figure 6 below for examples). the game requires cooperation, with the first player needing to form and name an ad-hoc concept that spans the target concepts, in a way that they assume is understandable to others (theory of mind) @xcite . players need to con-nect words in a wide variety of relations such as homonyms, antonyms, rhymes, or popular culture references @xcite . clue generation is also a task of co-creativity @xcite , testing skills in the evaluation of semantic relatedness of words and common-sense reasoning @xcite as well as the ability to constrain clues and negatively associate them with any non-team word. players need to predict the partners' behaviour and knowledge @xcite , so one cannot simply optimise their own behaviour @xcite without acknowledging the cultural background and knowledge level of their teammates @xcite , hence requiring cooperation. figure 1 shows the overall idea where players (spymaster and field operative) are llms that play against a programmed mock opponent. the programmatic gamemaster component (part of the framework we use) orchestrates the game play by providing inputs and generated outputs among the parties, checks whether players comply to the rules of the game. our contributions are as follows: i) benchmarking llms to test their ad-hoc concept generation, cooperation, pragmatic reasoning capabilities, ii) comparison of open-weight and commercial models under various experiments, iii) in-depth analysis of how best-performing models navigate the task. ## related work earlier work focused on using various word embedding techniques (choose the clue that is closest to targets and most distant to distractors) @xcite . later, such methods were combined with associative methods that use language graphs for generating clues or guessing @xcite . other approaches involve concentrating on word co-occurrence measurements (de @xcite for capturing synonymy, semantic similarity, or word-relatedness measure instead of just focusing on word embeddings. later research started looking into using llms to generate clues @xcite . the idea of benchmarking llms led to the development of various datasets, e.g. bigbench @xcite includes codenames as one of the many tasks to test emergent abilities of models @xcite @xcite . @xcite recently explored using codenames to benchmark llms where two pairs of llms (red vs. blue team) play against each other. our method differs from theirs by comparing the language model team against a deterministic opponent. not using a deterministic opponent could lead to different results every time the same game is played due to the non-deterministic nature of language models @xcite . another extension of our work lies in the experimental setup, where we study the effect of selecting words on a board and their relations in much more depth. ## the game: codenames codenames @xcite is a cooperative board game with two teams (blue and red) that try to uncover their team agents' code names before the other team finds all of theirs. the board is set up with 25 word cards. each team has a "spymaster" that knows which words on the board represent their team (8+1 for the starting team), the opposing team (8), innocent bystanders (7), and the assassin (1). the team starting the game has one more word to uncover to balance out the advantage of going first. our implemented version deploys one spymaster and field operative on the same team. the opponent team is mocked with an ideal behaviour of revealing n own words each turn. the spymaster takes turns providing clues for their teammates -the "field operatives". each clue consists of a word related to one or more (code names) targeted words. it has to output in this way: ## discussion commercial vs. open: we can notice that commercial models outperform open-weight ones by some margin. we categorised the errors by models and counted them (see ## conclusion we implemented codenames to benchmark llms by targeting their pragmatic reasoning, language understanding specifically for ad-hoc concept generation, and cooperation capabilities. we tested the most recent commercial and open-weight models on various experiments and difficulty levels. we can generally confirm that commercial models are ahead in performance compared to open-weight ones. the main reasons for better performance can be attributed to having less errors with regards to hallucinations, instruction following, and pragmatic reasoning. however, when looking at played episodes, we can say that even the best performing models do not win over 50% of the games. it clearly indicates that the task is far from being solved. overall, the presented solution provides a clear method for benchmarking llms using gamebased evaluation to target specific capabilities. model target hallucinated guess hallucinated rambling error repeated clue prefix error wrong # of guesses guess is clue repeated target o3-mini 0 0 figure 7 : average number of words that are targeted, guessed, and revealed for selected models. team words: wind, shadow, pupil, spider, lab, glove, shakespeare, card, press. opponent's team words: racket, rabbit, jupiter, flute, field, band, microscope, europe. distractor words: charge, snow, mercury. the kill words: bill, glass, lap, africa, chair. clue: black targets: wind, shadow, pupil, spider, lab, glove, shakespeare, card, press guess: pupil, card, glass, africa, lab, rabbit, spider, shadow, band clue: dark targets: shadow, spider guess: shadow, spider clue: experiment targets: lab, pupil guess: lab, mercury clue: student targets: pupil, shakespeare guess: pupil, chair lost "africa" is a kill word o3-mini lost "chair" is a kill word claude-3.5 clue: hand targets: glove, spider, card, pupil, shakespeare guess: glove, hand is not in the list but related words are, lap, band, card, racket aborted "too many guesses, allowed 5" llama-3.3-70b risk level -5 assassin words clue: theater targets: shakespeare, card guess: shakespeare, shadow clue: science targets: lab, pupil guess: lab, glass lost "glass" is a kill word gemini-2.0 clue: cover targets: glove, shadow, card, press guess: glove, press, snow, charge clue: black targets: shadow, pupil, spider guess: shadow, spider, pupil clue: paper targets: shakespeare, lab, card deepseek-r1 guess: bill, card, charge lost "bill" is a kill word distractor words guessed: charge, snow team words: page, race, peep, firm, spell, press, toll, grave, flat. opponent's team words: vault, sheer, peck, card, liver, rank, jig, gag. distractor words: pump, scrub, rag, stem, bowl, bull, nip. the kill words: bank game is won! o3-mini deepseek-v3 clue: counter targets: spell, press guess: spell, card clue: dead targets: race, toll, grave, flat guess: grave, flat clue: news targets: page, firm, press guess: press clue: call targets: page, peep, firm, toll guess: page, toll clue: people targets: race, peep, firm guess: peep, race clue: company targets: firm guess: firm clue: sound targets: peep, toll, press, grave guess: peep, toll, flat, sheer clue: print targets: page, press, spell guess: page, press, card opponent words guessed: card clue: competition targets: race, firm opponent words guessed: sheer guess: race, bull clue: magic targets: spell, grave guess: spell, vault opponent words guessed: vault clue: solid targets: firm, grave guess: firm, grave game is won! clue: letter targets: page, spell, press guess: page, spell, press clue: road targets: race, toll, flat deepseek-r1 lost! "bank" is a kill word guess: toll, race, bank