Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language (2021)

Abstract: Current NLP datasets targeting ambiguity can be solved by a native speaker with relative ease. We present Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced. Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a misleading surface reading, whose solving requires disambiguating semantic, syntactic, and phonetic wordplays, as well as world knowledge. Cryptic clues pose a challenge even for experienced solvers, though top-tier experts can solve them with almost 100% accuracy. Cryptonite is a challenging task for current models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6% accuracy, on par with the accuracy of a rule-based clue solver (8.6%).

## introduction the ambiguity of natural language is one of the most fundamental challenges in nlp research. while there are works and datasets specifically targeting ambiguity @xcite @xcite , these can be solved by a native speaker with relative ease. can we design a dataset with ambiguities that pose a challenge even to competent native speakers? we present cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced. cryptonite's 523k examples are taken from professionallyauthored cryptic crosswords, making them less prone to artifacts and biases than examples created by crowdsourcing @xcite . each example in cryptonite is a cryptic clue, a short phrase or sentence with a misleading surface reading, which poses a challenge even for humans experienced in cryptic crossword solving. a cryptic clue usually consists of two underlying parts: wordplay and definition. the one doesn't like shifting earth (5) hater wordplay defini�on synonym anagram enu. we provide a standard baseline by fine-tuning the generic t5-large conditional language model @xcite on cryptonite, achieving only 7.6% accuracy. for comparison, a rule-based cryptic clue solver (deits, 2021) achieves 8.6% accuracy. these results highlight the challenge posed by cryptonite, making it a candidate for assessing the disambiguation capabilities of future models. analyzing the results of both baselines, we find a correlation between performance on individual clues and a human assessment of the clue's difficulty, and that the enumeration (answer length) is highly informative. finally, we show that ensuring that the answers of train and test examples are mutually exclusive is critical for a candid estimation of t5's ability to solve cryptic clues in general. ## cryptic crosswords a cryptic crossword, just like a regular (noncryptic) crossword, is a puzzle designed to entertain humans, comprised of clues whose answers are to be filled in a letter grid. unlike regular crosswords, where answers are typically synonyms or hyponyms of the clues @xcite , cryptic clues have a misleading surface reading, and solving them requires disambiguating wordplays. a cryptic clue has only one possible answer, even when taken outside the context of the letter grid. 1 generally, a cryptic clue (henceforth, "clue") consists of two parts: wordplay and definition. the wordplay-definition split is not given to the solver, and parsing it is usually the first step in solving a clue. both the wordplay and the definition lead to the answer, but each in a different manner. while the definition is directly related to the answer (e.g. a synonym or a hypernym), the wordplay needs to be deciphered, usually with the help of an indicator that hints at the wordplay's type. figure 2 walks through several clues with different types of wordplays and their solution. these examples provide only a glimpse into the rich and diverse world of cryptic crosswords. for a deeper dive, see the guidebook by @xcite . ## dataset we introduce cryptonite, a dataset of 523,114 cryptic clues from 17,375 english-language crosswords published in the times foot_1 and the telegraph foot_2 between october 2000 and october 2020. 4 for preprocessing, we remove clue-answer duplicates and examples whose answer and enumeration do not match. in addition, we remove any examples with the same clue but with a different answer. while this occurred in less than 0.1% of the data, these examples violate the principle that a clue must have a single solution once the wordplay is deciphered (see section 2). ## experiments we provide initial results on cryptonite using two baselines: t5-large @xcite and a rule-based cryptic clue solver @xcite . despite training on half a million clues (t5) or being tailored to the task (rule-based solver), both approaches solve only a small portion of the test data, demonstrating that cryptonite is indeed a challenging task. we further investigate two properties of the data: how difficulty (as perceived by humans) correlates with accuracy, and the informativeness of enumeration. in addition, we analyze how a naive data split affects the performance of t5, demonstrating that partitioning by answers is crucial for obtaining a candid estimate of the neural model's ability to generalize to new cryptic clues. ## conclusion we presente cryptonite, a large-scale dataset based on cryptic crosswords, whose solving requires disambiguating a variety of wordplays. we saw that the standard approach of fine-tuning t5-large on cryptonite does not outperform an existing rulebased model, achieving 7.6% and 8.6% accuracy respectively, while human experts achieve close to 100% accuracy. these results highlight the challenge posed by cryptonite, and will hopefully encourage further research on disambiguation tasks that are not easily solved by a native speaker.