e P i C : Employing Proverbs in Context as a Benchmark for Abstract Language Understanding (2022)

Abstract: While large language models have shown exciting progress on several NLP benchmarks, evaluating their ability for complex analogical reasoning remains under-explored. Here, we introduce a high-quality crowdsourced dataset of narratives for employing proverbs in context as a benchmark for abstract language understanding. The dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and contains minimal lexical overlaps between narratives and proverbs, ensuring that models need to go beyond surface-level reasoning to succeed. We explore three tasks: (1) proverb recommendation and alignment prediction, (2) narrative generation for a given proverb and topic, and (3) identifying narratives with similar motifs. Our experiments show that neural language models struggle on these tasks compared to humans, and these tasks pose multiple learning challenges.

## introduction large language models (llms) @xcite @xcite @xcite @xcite have led to a paradigm shift in nlp, and have shown exciting progress on benchmarks such as glue @xcite and superglue @xcite . in particular, these include tasks such as reading comprehension, natural language inference, and coreference resolution. many of these tasks rely on semantic and syntactic reasoning, which has been mastered by these llms. for example, apart from improving on distributional semantics through contextualized embeddings @xcite , recent work has shown evidence that these models implicitly learn emergent concepts such as subject-verb agreement @xcite , semantic roles @xcite and dependency structures @xcite . prevention is better than cure instead of working on his english assignments weekly, he put them off until the last week before they were all due. he was able to finish them all over the final few days, but it took a lot of energy drinks and misery. there was once a disease that spread like wildfire. it killed people by the thousands. doctors said wash your hands regularly and you'll be ok. eventually an inoculation shot was created, and it worked, except on the people who refused to wash their hands and died waiting for their turn to get a shot. figure 1 : we introduce epic, a crowdsourced dataset of narratives for employing proverbs in context. our dataset contains narratives (n1 and n2) paired against proverbs (p) along with a fine-grained annotation of aligned spans between the narratives and proverbs. aligned spans are shown with matching colors and indicate correspondences in roles between proverbs and narratives. we explore three tasks: (1) proverb recommendation and alignment prediction (predict p given n1), (2) narrative generation for a given proverb and topic (generate n1 given p and k1), and (3) identifying narratives with similar motifs (e.g. identify n2 in a set of narratives given n1). however, humans show an ability for deeper linguistic reasoning. we can identify people's intentions and goals @xcite , perform relational reasoning @xcite , and find analogies in situations with little surface overlap @xcite . in particular, making verbal analogies in the form of proverbs is noted as an indicator of literary ability @xcite . proverbs are also repositories of information on culture, societal norms, values, and folk wisdom @xcite . in this work, we investigate proverbs in narrative contexts as a testbed for evaluating abstract reasoning and analogical abilities of llms. we introduce epic (employing proverbs in context), a high-quality crowdsourced dataset of narratives paired with proverbs. the dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and is designed to minimize lexical overlap between narratives and proverbs. figure 1 shows two examples of narratives for a proverb from our dataset, along with corresponding alignment annotations. we diverge from related extant resources @xcite @xcite on using proverbs in terms of quality of narratives, direct supervision, and having fine-grained alignment annotations. 1 we explore three tasks: (1) proverb and alignment prediction ( § 5.1), (2) narrative generation for a given proverb and a set of keywords specifying a topic ( § 5.2), and (3) discovering narratives with similar motifs ( § 5.3). by benchmarking several llms, we find that existing models struggle with these tasks, suggesting much scope of improvement in abstract reasoning. in particular, humans show much higher performance in many cases. in §3, we describe the crowdsourced creation of the epic dataset. in §4, we analyze lexical overlap, biases, and narrative quality in epic. §5 describes the three tasks and details of experimental evaluation of llms for each task. we conclude with a discussion, and a statement of ethics and broader impact relevant to our work. our contributions are: • we introduce epic, a high-quality dataset for employing proverbs in context. it contains multiple narratives for english proverbs and fine-grained annotation of aligned spans between them. our dataset and code are publicly available at: https://epic-benchmark.github.io ## related work prior works in figurative language understanding have explored a diverse set of topics, such as simile detection and generation @xcite @xcite , metaphor detection and generation @xcite @xcite @xcite , pun identification @xcite , and quote/proverb recommendation @xcite @xcite . recent work @xcite has also focused on interpreting idioms and similes in narratives. @xcite has explored recommending chinese idioms through context-based recommendation and @xcite formulated idiom recommendation as cloze-style reading comprehension task. learning to quote has been explored based on fiction @xcite and noisy social media conversations from twitter, reddit or weibo @xcite . in the most related prior work, authors explore a quote retrieval task borrowing inspiration from context based recommendation systems @xcite . @xcite formulated learning to quote as a generation task by using topic modeling @xcite in a sequence-to-sequence network. while previous work has considered idioms, proverbs and common phrases as quotes, we specifically work with proverbs. compared to earlier datasets, our dataset is manually created and labeled. further, epic includes fine-grained annotations aligning parts of proverb to parts of the narrative, which has significant possibilities for model training, evaluation and interpretability. ## dataset creation in this section, we describe the steps involved in creating the dataset in detail. proverb collection: we obtained a candidate set of english proverbs by scraping websites of 'the phrase finder' 2 and wikiquotes 3 . next, this set was manually pruned to remove lexical variations of the same proverb. this manual curation led to a set of 250 proverbs, which we consider in the current version of our dataset. narrative collection: in the second step, we use amazon mechanical turk to collect a diverse set of narratives corresponding to each proverb. we collect 10 narratives contributed by distinct turkers for each proverb, leading to a total of 2500 proverbnarrative pairs. we also ensure that no turker contributes a large number of narratives to alleviate annotator bias @xcite (where models can overfit to annotator characteristics) while encouraging diversity in writing style and content. the turkers were asked to write short realistic stories, preferably within 100 words. additionally, to avoid surface-form biases, turkers were encouraged to minimize lexical overlap and to not mention the proverb or parts of it in the narrative. this was done so that doing well on the tasks requires a detailed understanding of the narratives rather than relying on surface-level cues. turkers were paid 50 cents for each narrative for this task. span alignment annotation: next, we solicit finegrained annotations between the narratives and the proverb in form of aligned spans. for this, we present proverb-narrative pairs to turkers asking them to find contiguous spans in the narrative which align well with contiguous spans in the proverb. turkers could submit up to 5 pairs of aligned spans per proverb-narrative pair. these aligned spans highlight the grounding of a proverb in the narrative (see figure 1 ). these annotations can help to verify the reasoning capabilities of various neural models by checking if these models are able to identify these correspondences, and add interpretability to our tasks. turkers were paid 25 cents for each proverb-narrative pair annotation for this task. statistics: ## dataset analysis ## tasks & evaluation in this section, we introduce three tasks associated with epic and describe their experimental setup and benchmark results: (1) proverb and alignment prediction, (2) narrative generation, and (3) identifying narratives with similar motifs. ## conclusion and future work we introduce epic, a high-quality crowdsourced dataset of narratives paired with proverbs, and a suite of challenging tasks associated with this dataset. we show that these provide a challenging testbed for evaluating abstract reasoning and analogical abilities of llms. future work can explore more sophisticated mechanisms to use alignment annotations in improving the performance for proverb prediction and model interpretability. additionally, researchers can explore conditional narrative generation through more informative prompts than using keywords. epic can also be extended in the future by incorporating more proverbs and adding more layers of complexity like sarcasm or adversarially creating harder narratives. most of all, the development of similarly challenging resources and tasks can enable the possibility of socially grounded nlp systems.