Language Models are Crossword Solvers (2025)

Abstract: Crosswords are a form of word puzzle that require a solver to demonstrate a high degree of proficiency in natural language understanding, wordplay, reasoning, and world knowledge, along with adherence to character and length constraints. In this paper we tackle the challenge of solving crosswords with large language models (LLMs). We demonstrate that the current generation of language models shows significant competence at deciphering cryptic crossword clues and outperforms previously reported state-of-the-art (SoTA) results by a factor of 2-3 in relevant benchmarks. We also develop a search algorithm that builds off this performance to tackle the problem of solving full crossword grids with out-of-the-box LLMs for the very first time, achieving an accuracy of 93% on New York Times crossword puzzles. Additionally, we demonstrate that LLMs generalize well and are capable of supporting answers with sound rationale.

## introduction modern llms demonstrate astounding skills in reasoning, coding, wordplay, question answering (qa), and a multitude of other tasks @xcite . despite the plethora of applications for llms seen today, their ability to generate language in a constrained setting remains relatively uncharted, and the ability to direct the generation process in order to meet certain 1 code, data, etc., can be found at https://www.github. com/espressovi/lmcrossword criteria remains a challenge @xcite . consider the task of poem generation as an example, where, in addition to thematic aptness, constraints on rhyme or meter must be adhered to. further, for certain kinds of poems like haikus, sonnets, or even song lyrics, restrictions on length, syllable counts, or patterns of stressed/unstressed syllables apply. constraints also arise when dealing with formal languages @xcite , which is increasingly relevant in the llm zeitgeist given their use in coding or interacting with databases, interpreters, apis, etc. with a growing body of literature studying llms as agents @xcite , language generation must follow environmental or physical constraints. for wider proliferation of llm applications, they must demonstrate the ability to adhere to such domain-specific constraints (e.g., constraints imposed by knowledge graphs, tabular data, action spaces, etc.). solving crosswords requires proficiency in understanding contextual clues, semantics, wordplay, character manipulation, arithmetic, and reasoning (see figure 1 (right)), along with satisfying constraints such as length limitations and character overlaps (see figure 1 ). given the multi-faceted nature of this task, it can serve as a testbed for studying constrained language generation. since solving crosswords requires proficiency in several desirable areas, and identifying shortcomings can readily benefit other linguistic applications, in this paper we attempt to analyze llms' ability to solve crossword puzzles, with the primary goal of understanding strengths and weaknesses demonstrated by sota llms. our contributions in this study are as follows: we believe our work has been a major step towards demonstrating the power of llms with regard to crossword solving, and with future advances in certain key areas, llms' performance in this task will be comparable to, if not better than, human experts and specialized systems. our algorithmic approach of combining an llm with a search strategy might be extended to linguistic problems with domain-specific constraints. ## background traditional approaches to solving straight crossword puzzles involve two key components: a candidate answer proposal system and a grid-filling algorithm @xcite . answer proposal systems typically use similarity search on large clue-answer databases, finetuned language models, or a combination of both @xcite . gridfilling relies on variants of constraint satisfaction problem (csp) algorithms. for instance, the system @xcite achieves a 98.1% letter accuracy on new york times (nyt) crosswords. @xcite fine-tuned bert and byt5 on 6.4 million clue-answer pairs, and using a belief propagation algorithm, achieved a 99.7% letter accuracy. @xcite established benchmarks with foundational language models and highlighted this task as "... a new high bar for ai systems. ". note that this work does not attempt to create an improved automated crosswordsolving system but seeks to analyze foundational llms' ability at this complex language task. cryptic crosswords are more formidable and involve extensive wordplay such as anagrams, splicing, homophones, and puns. traditional algorithms with large clue datasets and a context-free grammar parser (deits, 2015) have shown poor performance, achieving only 7% accuracy @xcite . recent studies have explored using large language models (llms) to solve cryptic crossword clues. @xcite created a large dataset of cryptic crossword clues from uk dailies, the times and the telegraph, and fine-tuned a t5-large @xcite model to establish baseline performances. they used a training split where answers in the training and test sets were mutually exclusive to prevent memorization. @xcite curated a dataset from the guardian and fine-tuned a t5-large model using curriculum learning, showing performance improvements. they critiqued @xcite 's approach, arguing that a disjoint train-test split is insufficient for teaching models to solve cryptic crosswords, as models exhibit "... robustness to plural and other inflections." instead, they proposed grouping similar-root words together in a split, noting that this more stringent criterion led to reduced performance. the most recent work by sadallah et al. (2024) presents results with recent llms like mistral-7b @xcite , llama2-7b @xcite , and chatgpt (openai, 2021) in few-shot settings and also by fine-tuning the mistral model. they report that chatgpt surpasses other models with an accuracy of 9.5%, which demonstrates a significant performance gap with respect to human experts, who solve 99% of cryptic crossword clues @xcite . they point out key limitations in their work, like the limited set of llms used and the potential for data contamination. these recent works on cryptic crossword solving with llms highlight a significant performance gap between llms and human experts. however, these recent works approach the problem as a question answering (qa) task and ignore constraints imposed by the grid. this needs to be investigated further since it is yet to be seen whether a suitable approach that integrates constraint information into llms can yield significant performance benefits. as for straight crosswords, @xcite attempted to solve nyt crossword puzzles with language models and a satisfiability modulo theory (smt) solver with limited success, and resorted to culling the crossword grid based on candidate generations and ground truth answers. we present an algorithm employing llm-guided search, which is significantly more successful at this task. with advancements in llm capabilities and methods like ours, we believe llms could soon outperform humans in solving cryptic crosswords too. ## analyzing crossword clue solving the first step in solving a crossword puzzle is deciphering its clues, so following in the scheme of @xcite , @xcite , and @xcite , in this section, we first explore this as a qa task. we analyze the performance of several llms at different scales with variations of this task. ## sweepclip -our proposed algorithm to solve crosswords with llms in this section, we address the problem of filling crossword grids with llm assistance. note that this task not only involves generating correct answers from provided clues but also hinges on exploiting constraints from already generated words and backtracking to eliminate past generations that do not fit well when new evidence becomes available. since llms demonstrate this ability, when paired with the right search algorithm, it should be possible to solve crosswords with the aid of llms. our proposed algorithm (detailed in appendix a) first generates a set of candidate answers for all clues provided with the crossword (sweep) and uses a graph-based criterion to eliminate answers that do not fit (clip). then we use the constraints generated from the previous step answers to generate more candidate answers foot_4 and prune the bad-fitting candidates. we iteratively apply this strategy until either (i) the entire crossword is filled, (ii) we exceed a preset number of iterations, or (iii) we run out of llm computational budget. for pruning, we use the largest connected component from the answers generated so far to ensure that the generated answers are somewhat coherent amongst themselves. this is over-restrictive, as it is possible that isolated answers are correct; however, we find that this strict pruning strategy works better to eliminate bad answers early rather than using the constraints imposed by potentially bad answers to generate further bad answers (see appendix a). ## results in this section, we present results from our algorithm at the crossword-solving task and present further results investigating the performance demonstrated by current-generation sota llms. ## conclusions solving crosswords requires proficiency in a multitude of desirable skills, and cryptic crosswords were previously thought to be firmly in the domain of human experts. our findings challenge this notion. we have demonstrated that the current generation of sota llms shows significantly improved aptitude at solving straight and cryptic crossword puzzle clues and can exploit constraints provided by partially solved crosswords to boost this performance further. we also found that this emergent ability generalizes well to the post knowledge cutoff regime and is accompanied with the capacity to produce reasoned explanations. we have also developed an algorithm that, with the aid of llms, further boosts this performance and achieves 93% accuracy on monday new york times crosswords. with the aid of this algorithm, it is possible to achieve significant performance gains even if the baseline llm accuracy is low, thus indicating that, when paired with the right search strategy, llms can successfully solve cross- incorporating length constraints effectively still remains a challenge. we showed that sota llms struggle with sub-token counting and also that they often provide incorrect predictions despite being semantically close, owing to length constraint violations. this weakness also manifests in other tasks like anagrams, character manipulations, etc., which are heavily reliant on arithmetic abilities. future research attempting to address this shortcoming would be compelling.