Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLM s Using the N ew Y ork T imes Connections Word Game (2024)

Abstract: The New York Times Connections game has emerged as a popular and challenging pursuit for word puzzle enthusiasts. We collect438 Connections games to evaluate the performance of state-of-the-art large language models (LLMs) against expert and novice humanplayers. Our results show that even the best-performing LLM, Claude 3.5 Sonnet, which has otherwise shown impressive reasoning abilities on a wide variety of benchmarks, can only fully solve 18% of the games. Novice and expert players perform better than Claude 3.5 Sonnet, with expert human players significantly outperforming it. We create a taxonomy of the knowledge types required to successfully cluster and categorize words in the Connections game. We find that while LLMs are decent at categorizing words based on semantic relations they struggle with other types of knowledge such as Encyclopedic Knowledge, Multiword Expressions or knowledge that combines both Word Form and Meaning. Our results establish the New York Times Connections game as a challenging benchmark for evaluating abstract reasoning capabilities in humans and AI systems.

## introduction abstract reasoning represents a person's ability to solve problems, identify patterns, and work with logical systems @xcite @xcite . while the performance of large language models (llms) on arithmetic and language-based commonsense reasoning benchmarks has been the subject of recent analyses, it is unclear whether these llms possess abstract reasoning capabilities that are often challenging even for humans @xcite . we propose the nyt connections game as a test bed for investigating the abstract reasoning capabilities of both humans and large language models (llms). connections is an engaging game launched by the new york times (nyt) in june 2023. this daily game presents players with a 4x4 grid containing 16 words and tasks them with identifying four distinct clusters that link the corresponding four words in each cluster through some shared characteristics @xcite ). categories 1 (yellow), 2 (green), 3 (blue), and 4 (purple) are arranged in ascending level of difficulty. category 1 is the most intuitive, while category 4 is the hardest. for instance, in figure 1 (b) , the most straightforward category is "conformists" {followers, lemmings, puppets, sheep}, while the most challenging category includes {apartment, insults, likes, shovels} and requires the understanding that a single word (in this case, "digs") can have multiple meanings that differ in etymology or sense, depending on the context. while the task might seem easy, many words can be grouped easily into multiple categories, acting as red herrings. for instance, from the game in figure 1 , likes, followers, shares, insult might be categorized as "social media interactions" at first glance. unlike common categories (e.g., "fruit," "furniture"), the game is designed to promote ad hoc category formations that violate the correlational structure of the environment and are not well established in memory @xcite . to group words across proper categories, as shown in figure 1 (b), a player must reason with various forms of knowledge spanning from semantic knowledge (conformists) to encyclopedic knowledge (u.s. cities). we test the capabilities of five state-of-the-art large language models, namely google's gemini 1.5 pro @xcite ), anthropic's claude 3.5 sonnet @xcite , openai's gpt-4omni (openai, 2023), meta's llama 3.1 405b, (ai@meta, 2024) and mistral large 2 (mistral-ai, 2024) on 438 distinct nyt connections games and compare them with human performance on a subset of these games. our experimental results show that while all llms can partially solve some games, their performance is far from ideal. even the bestperforming model claude 3.5 sonnet (with fewshot and chain-of-thought prompting), can only solve 18% of the games completely. in addition, we recruit human players at novice and expert levels of proficiency and compare their performance to claude 3.5 sonnet. our results show that the nyt connections game serves as a challenging benchmark for reasoning, with novice players performing marginally better than claude 3.5 sonnet and expert players performing significantly better than claude 3.5 sonnet in solving games perfectly (section 5). in addition, to better understand the llms abstract reasoning capabilities or the lack thereof, we propose a taxonomy of knowledge required to group words into their respective categories (section 3.2). our analysis at section 6.1 shows that while llms are relatively better at reasoning that involve semantic relations, they struggle with other types of knowledge such as multiword expressions and combined knowledge about word form and word meaning (section). our code and data will be made available to the public at https://github.com/mustafamariam/ llm-connections-solver . ## related work the growing popularity of large language models has led to an exciting array of research using natural language processing techniques for text-based games. recent work has studied whether these models can act as players in agentic environments @xcite @xcite or conversational settings @xcite . in addition to acting as players, researchers have also tested the abilities of transformer-based language models in generating games @xcite @xcite @xcite recent research has explored applying large language models (llms) and other natural language processing (nlp) techniques to solve and generate text-based puzzles. @xcite propose automatic ways of solving crossword puzzles by generating answer candidates for each crossword clue using neural question answering models and combining loopy belief propagation with local search to find full puzzle solutions. @xcite release puzzleqa, a multiplechoice dataset comprising 15 years of on-air sunday puzzle word-games and show that while chat-gpt can solve these questions with an accuracy of around 50%, they still struggle with generating novel and engaging puzzles. unlike the nyt connections game, puzzleqa relies on characterlevel word transformations compared to encyclopedic, associative, or semantic knowledge. @xcite examined the potential of using "cryptic crossword" clues as an nlp benchmark. most relevant to our paper is the contemporaneous work by @xcite who test the performance of various llms (from bert and roberta to gpt-4 and gpt-4 with chain-of-thought prompting) in solving the nyt connections game. our work is similar in that it utilizes the nyt connections puzzles as a means to investigate the abstract reasoning capabilities of state-of-theart llms. however, our contribution focuses not only on the ability of llms to solve the game, but also on studying the types of knowledge required to so. moreover, we evaluate both state-of-theart models (open and closed weights) and humans (novices and experts). we also benchmark on a larger number of nyt connections game. additionally, @xcite 's experimental setup mirrors the way the original nyt connections game is played (i.e., one category at a time, with an allotment of 4 incorrect guesses), while we require both humans and llms to provide all categories at once in only one attempt. the word association task @xcite has been used extensively in psychological and linguistic research as a way of measuring connections between words in the mental lexicon. responses in word association tasks have informed what we know about the structure and organization of semantic memory and the mental lexicon @xcite . in this work, we similarly show how one must utilize semantic and associative memories to solve the nyt connections game. @xcite proposed the abstraction and reasoning corpus (arc), built upon an explicit set of priors designed to be as close as possible to innate human priors and argued that it can be used to measure a human-like form of general fluid intelligence, enabling fair general intelligence comparisons between ai systems and humans. recently @xcite show that gpt-4 solves only 13/50 of the most straightforward arc tasks, demonstrating a significant gap in the abstract reasoning capabilities of llms. prior work has also studied abstract reasoning in neural networks @xcite in the presence of distracting features @xcite . our work builds upon these and presents the nyt connections game as a compelling benchmark for abstract reasoning capabilities of llms in the presence of distractors. ## limitations many of the limitations in this section stem from the lack of data available fornyt connections games and disparities in the comparison between llms and humans. because it is a fairly recent invention and only one puzzle is released per day, there are only a few hundred games available. since there are some category patterns learned through frequent play, ideally, a model trained on past nyt connections games might bridge the gap between llm and expert human evaluators' performance. we acknowledge that human evaluators were not required to add justifications for the groupings they made. this might have made performance comparisons between humans and llms for the types of reasoning more equal. additionally, because a score of 3 was impossible in the human evaluation interface, we cannot be certain that humans were adept in the type of knowledge of their last category grouped, as this could simply have been a matter of grouping all options left. other limitations of human evaluators include that because they were all peers or acquaintances of the paper's authors, sampling bias could exist. though the age range of the humans recruited was 14-60, other demographic factors that not have been accounted for in this sample. release. besides this, we ensure that now and in the future, any data collection is transparent with users and is used in an ethical and responsible manner. since our research primarily evaluates reasoning in a game environment, there are fewer potential realworld risks of its applications. however, biases in llms may be reproduced.