Evaluating Cultural Knowledge and Reasoning in LLM s Through P ersian Allusions (2025)

Abstract: Allusion recognition—a task demanding contextual activation of cultural knowledge—serves as a critical test of LLMs’ ability to deploy stored information in open-ended, figurative settings. We introduce a framework for evaluating Persian literary allusions through (1) classical poetry annotations and (2) LLM-generated texts incorporating allusions in novel contexts. By combining knowledge assessments, multiple-choice tasks, and open-ended recognition, we analyze whether failures stem from knowledge gaps or activation challenges. Evaluations across eleven LLMs highlight a notable observation: models exhibit strong foundational knowledge and high multiple-choice accuracy, yet performance drops substantially in open-ended tasks, especially for indirect references. Reasoning-optimized models generalize better to novel contexts, whereas distilled models show marked degradation in cultural reasoning. The gap underscores that LLMs’ limitations arise not from missing knowledge but from difficulties in spontaneously activating cultural references without explicit cues. We propose allusion recognition as a benchmark for contextual knowledge deployment, highlighting the need for training paradigms that bridge factual recall and culturally grounded reasoning. Our code, datasets and results are available at https://github.com/MelikaNobakhtian/Allusion

## introduction allusion-the indirect reference to a culturally or historically significant entity-presents a unique challenge for both human readers and language models. recognizing an allusion requires more than surface-level comprehension: it demands retrieving culturally situated background knowledge and applying it in novel contexts. this makes allusion recognition an ideal benchmark for evaluating model recall: the ability to access and deploy while large language models (llms) excel in factual recall and generalization, their ability to activate knowledge in open-ended, figurative settings remains underexplored. prior work has critiqued multiple-choice (mc) formats for llm evaluation and advocated for open-ended tasks to assess reasoning @xcite , with some proposing the conversion of open-ended questions into an mc format for efficiency @xcite . however, these investigations primarily target numerical or logical tasks, leaving figurative language, particularly allusions, largely understudied. unlike metaphors or idioms @xcite @xcite , allusions are less formulaic and demand recognition of indirect, culturally embedded references, making them a rigorous test of cultural reasoning. despite the potential, limited work has focused on allusions as a primary task for evaluating llms @xcite . we introduce an evaluation framework for allusion recognition in persian literature, a tradition rich in symbolic and indirect references. we construct two datasets: (1) a dataset of 200 annotated samples of classical persian poetry (perspoems), and (2) a dataset of 75 llm-generated allusive texts incorporating the same allusions in novel outof-distribution contexts. these datasets disentangle knowledge from memorization, probing llms' ability to recognize allusions in unfamiliar settings. our dual framework integrates mc tasks, which isolate discriminative skills, and open-ended recognition tasks, which test spontaneous knowledge activation, complemented by a knowledge assessment covering 127 core allusions. our results reveal a critical discrepancy: while llms demonstrate strong factual knowledge of allusions (e.g., correctly identifying referenced entities), they struggle to activate this knowledge in open-ended tasks, with performance dropping sharply compared to mc settings. reasoningoptimized models show more consistent performance across datasets, indicating better integration of cultural reasoning. this gap between knowledge and recognition suggests that recall failure, and not knowledge deficiency, limits llms' interpretive capabilities, highlighting the challenges of applying contextual knowledge in figurative and culturally embedded settings. ## allusion datasets here, we describe the two datasets used in our experiments: a collection of persian poems containing allusions and a set of allusive llm-generated texts created to test allusion recognition capabilities beyond potential training data memorization. detailed construction and annotation procedures are provided in appendix a. ## evaluation we evaluate llms on the allusion recognition task in three different settings: knowledge assessment, mc recognition, and open-ended recognition. ## results we evaluated 11 open-and closed-source llms for knowledge assessment and allusion recognition on both the perspoems dataset and the llm-generated dataset in open-ended and multiple-choice settings. our evaluation includes six open-source models: llama-3.3 (ai@meta, 2024), gemma-3 @xcite the connection. this pattern suggests that, in the absence of explicit narrative cues or the prompting effect of multiple-choice options, models struggle to activate relevant knowledge frameworks. similarly, claude-3.5-sonnet exhibits a common pattern across many llms in allusion recognition when handling allusions derived from quotations, quranic verses, or hadiths. while the model readily identifies such allusions when options are provided, it frequently fails in open-ended scenarios, particularly when the allusive text lacks explicit markers or conventional framing devices that signal a quotation or reference. an intriguing complementary pattern emerges in the opposite scenario: models with limited knowledge nonetheless achieve relatively high performance on multiple-choice tasks through surfacelevel lexical matching strategies rather than genuine comprehension. we provide a detailed analysis with representative examples in appendix b. ## conclusions we introduced two persian allusion datasets, per-spoems and llm-generated datasets, designed to probe llms' cultural reasoning beyond memorization. using multiple-choice and open-ended formats, we assessed both discriminative ability and spontaneous knowledge activation. we did our evaluation on six open-source and five closedsource llms. most llms showed strong factual knowledge and high accuracy in multiple-choice questions, but performance drops significantly in the open-ended setting. this gap reveals that recall failure (and not lack of knowledge) limits interpretive understanding. we also found that llms post-trained for reasoning using rl generalize better to our llm-generated data, pointing to the need for training and evaluation methods that support contextual cultural inference.