LLM s Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction (2024)

Abstract: In this study, we explore in-context learning (ICL) in document-level event argument extraction (EAE) to alleviate the dependency on large-scale labeled data for this task. We introduce the Heuristic-Driven Link-of-Analogy (HD-LoA) prompting tailored for the EAE task. Specifically, we hypothesize and validate that LLMs learn task-specific heuristics from demonstrations in ICL. Building upon this hypothesis, we introduce an explicit heuristic-driven demonstration construction approach, which transforms the haphazard example selection process into a systematic method that emphasizes task heuristics. Additionally, inspired by the analogical reasoning of human, we propose the link-of-analogy prompting, which enables LLMs to process new situations by drawing analogies to known situations, enhancing their performance on unseen classes beyond limited ICL examples. Experiments show that our method outperforms existing prompting methods and few-shot supervised learning methods on document-level EAE datasets. Additionally, the HD-LoA prompting shows effectiveness in other tasks like sentiment analysis and natural language inference, demonstrating its broad adaptability.

## introduction document-level event argument extraction (eae) aims to transform unstructured event information from documents into structured formats encapsulating event arguments, facilitating their interpretation and application in various domains @xcite . the prevalent approach for this task relies on the collection of labeled data and the subsequent model training via supervised learning @xcite @xcite @xcite . while effective, this approach comes with the significant drawback: it necessitates a substantial amount of training data, which is particularly burdensome and costly given the complexity inherent to document-level eae. in this context, in-context learning (icl) @xcite @xcite , an emergent ability of large language models (llms), offers a promising alternative to supervised learning. icl alleviates the need for large-scale data as it only uses a few examples as input-output pairs of the prompt to guide llms in performing the task on an unseen example. however, applying icl to document-level eae presents numerous challenges. the icl performance is highly sensitive to the design of in-context demonstrations, such as the selection of examples and the formatting of reasoning steps @xcite @xcite . consequently, several crucial challenges emerge concerning the prompting strategy: 1) example selection challenge. selecting optimal in-context examples for icl is pivotal, yet the understanding of what llms learn from these examples remains largely under-explored @xcite . this gap leads to a lack of systematic guidelines, resulting in a disorganized and inefficient example selection process. ## heuristic-driven demonstration construction building on our understanding of heuristic learning during icl, we aim to address the challenge of example selection for icl. experiments in section 2 indicates that heuristics are crucial for icl performance of llms, yet they are implicitly conveyed through explanations of examples. this implicitness complicates the examination of whether icl demonstrations contain diverse heuristics and leads to uncertainty about whether llms have recognized these heuristics. additionally, when solving a task, humans possess the ability to not only learn from examples but also learn from heuristics for efficient and accurate inference @xcite . this leads us to question whether llms can similarly leverage explicit heuristics to improve icl performance. therefore, we are motivated to explicitly providing llms with task-specific heuristics. our approach is illustrated below: replacing examples with explicit heuristics: diverging from traditional prompting strategies that construct prompt with examples where heuristics are implicitly embedded, we propose to replace most examples in the prompt with distinct taskspecific heuristics, as demonstrated by the heuristics in figure 5 . retaining minimum examples: a minimal number of examples are preserved to (1) illustrate the formatting of target task and reasoning steps, such as one example is required to illustrate the format of our link-of-analogy prompting, and (2) ensure a balanced coverage of labels in prompt to avoid introducing label bias. specifically, for documentlevel eae task, a single example is maintained to demonstrate the reasoning format. ## link-of-analogy prompting we propose the link-of-analogy prompting to address the challenges below: first, the eae task is characterized by its extensive variety of argument roles and event types, often exceeding a hundred, yet icl examples can only cover a very limited subset. this discrepancy raises a critical challenge: designing a prompting strategy that effectively addresses unseen event types. notably, the issue of handling unseen classes beyond limited icl examples is a prevalent problem in various nlp tasks. additionally, to concretize heuristic generation process, we provide heuristics for a specific argument, giver, within the prompt. this leads to the question of how to extend giver heuristics to other argument roles. finally, as highlighted in the introduction, applying cot prompting to non-reasoning tasks tends to degrade the step-by-step analysis into a one-step rationale @xcite , necessitating more proper prompting strate-gies for such tasks. inspired by the analogical reasoning @xcite , a core mechanism of human cognition, we seek to resolve the challenges presented. humans often understand a new situation by drawing an analogy to a familiar situation. for example, students often solve new problems by mapping solutions from known problems @xcite . similarly, we anticipate that llms will be able to extract information of unseen events or generate heuristics for unseen argument roles by drawing an analogy to events and heuristics provided in in-context examples. empirically, we find that llms are indeed capable of doing analogical reasoning when prompted appropriately. for example, when provided with the heuristic for giver in the prompt: "[giver] is the person, group, or organization in the document that gives the grant or gift", llms can make an analogy and generate the heuristic for the argument vehicle in the target question: "[vehicle] is the means of transport used to move the person or object". to further enhance the analogical reasoning capabilities of llms, we introduce our link-ofanalogy (loa) prompting strategy, which emulates the analogical reasoning process of human. cognitive science studies reveals that humans perform analogical reasoning through a sequence of retrieval, mapping, and evaluation @xcite . in alignment with this process, our method involves the same steps. specifically, in the retrieve step, given the base argument role r b , a set of heuristics @xmath0, h 2 , • • • , h k } for identifying r b , a target question and a target argument role r t , the llm will select the most suitable heuristic h b from h for identifying r t . in the mapping step, the llm employs analogy mapping r b : h b :: r t : h t to deduce the heuristic h t for r t . the llm then infers the argument a t of the target role based on the heuristic h t . finally, in the evaluation step, the llm will reassess the identified argument a t . this methodology is exemplified in the in-context example presented in figure 5 . ## understanding why hd-loa prompting works following the empirical validation of the effectiveness of our hd-loa prompting, this section delves into an analysis to elucidate why our method works. analysis of heuristic-driven demonstration construction method: firstly, our method naturally incorporates diverse distinct heuristics in prompt. as shown in section 2.2, inclusion of diverse heuristics can significantly boost the icl performance. in addition, cognitive research finds that humans use heuristics as efficient cognitive pathways to achieve more accurate inferences compared to complex methods @xcite . paralleling this human cognitive strategy, we enable llms to learn from explicit heuristics to enhance inference. specifically, for llms demonstrating suboptimal performance with standard prompting and in non-reasoning tasks where definitive rationales are elusive, the provision of explicit heuristics offers llms helpful strategies to use and enhance inference. moreover, as discussed in section 2, llms use implicit heuristics embedded conventional prompts to facilitate inference. by converting these implicit heuristics to explicit heuristics offers a more straightforward way to utilize heuristics and may potentially simplify the utilization of heuristics by llms. analysis of the link-of-analogy prompting: the loa prompting, which is inspired by the analogical reasoning of human cognition, enables llms to process new situations by drawing analogies to known situations. this ability is particularly useful in icl, where llms are always facing unseen samples and unseen classes. as evidenced by experiments in appendix c, the loa prompting is indeed effective in enhancing icl performance for classes unseen in the prompt. 7 related work document-level eae existing document-level eae studies are mostly based on supervised learning methods, which relies on the extensive collection of labeled data (ma et al., 2022; pouran ben veyseh et al., 2022; zhou and mao, 2022; xu et al., 2021; ebner et al., 2020; du and cardie, 2020a). only agrawal et al. (2022) exploits adopting llms on clinical eae though standard prompts that not involve any reasoning strategies. considering the potential of icl to reduce the dependency on large-scale labeled datasets and the revolutionize impact of llms, it is lack of study on prompting strategy tailored for the eae task. in-context learning icl enables llms to perform a target task by feeding a few prompted examples as part of the input @xcite . as the mechanism of icl is fundamentally different from supervised ml, the working mechanism of icl remains an open question @xcite . few studies have conducted preliminary explorations: @xcite showed that the label space, input text distribution and overall format contribute to the icl performance. @xcite concluded that examples that are semantically similar to the test sample are more effective. @xcite found that transformer-based icl can implement standard finetuning implicitly. in this work, we further hypothesize and validate that llms learn task-task specific heuristics from examples via icl. moreover, the performance of icl is very sensitive to example selection @xcite and the optimal selection criteria remains unclear. various studies proposed different ways: selecting examples based on complexity @xcite , mutual information @xcite , diversity @xcite , labeled dataset @xcite , etc. in this work, we convert the indiscriminate example selection process into a methodical approach that emphasizes task heuristics, making the example selection process more transparent. ## conclusion in this work, we hypothesize and validate that llms learn task-specific heuristics from demonstrations during icl, which can provide a guidance and simplify the example selection process. building upon this hypothesis, we introduce an explicit heuristic-driven demonstration construction strategy, and propose a link-of-analogy prompting method. these methods shed light on the heuristic learning of llms and the challenge of handling unseen classes in icl. extensive experimentation demonstrates the effectiveness and adaptability of our hd-loa prompting.