How do LLM s’ Preferences Affect Event Argument Extraction? CAT : Addressing Preference Traps in Unsupervised EAE (2025)

Abstract: Large Language Models (LLMs) have significantly improved the performance of unsupervised Event Argument Extraction (EAE) tasks. However, LLMs’ inherent preferences severely hinder their effectiveness in EAE, leading to what we term preference traps, namely, the Prior Knowledge Trap, the Sycophancy Hallucination Trap, and the Output Contradiction Trap. Existing approaches often fall into these traps due to misalignments between their prior knowledge, instructions, or output constraints and LLMs’ preferences, which significantly limits further performance gains. To address this issue, we propose Choose-After-Think (CAT), an unsupervised EAE framework designed to handle these preference traps through targeted measures. CAT innovatively divides the EAE task into two phases: identification of event information (argument roles) (Think Phase) and selection of the final answers from a candidate set (Choose Phase). This two-phase approach reduces the impact of individual token probability anomalies and ensures the integrity of EAE results. Experimental results demonstrate that CAT (based on the local 7B model, zero-shot setting) matches the performance of the best DeepSeek-R1 API model, with a significantly lower time cost.

## introduction event argument extraction (eae), which aims to extract structured event information (arguments and their roles) from event texts, is a critical and highly challenging task in information extraction (ie). to address the limitations of supervised eae, such as high training costs and poor model generalization, researchers have increasingly turned to unsupervised frameworks based on large language models (llms), achieving significant progress @xcite . however, llms' preference traps in unsupervised eae tasks severely degrade performance. current mainstream approaches-including prompt engineering @xcite , chainof-thought reasoning @xcite @xcite , and outputting in programming languages @xcite )-only address some of these traps while succumbing to others (see detailed analysis in section 5). just as many works have demonstrated llms' preferences and their impacts across various tasks @xcite @xcite , we summarize three preference traps in unsupervised eae tasks that severely affect llms' performance, as shown in figure 1 . prior knowledge trap: llms heavily rely on prior knowledge to identify event information, but lowquality knowledge often misleads them into ignoring or misidentifying arguments. sycophancy hallucination trap: llms often generate answers that conform to users' viewpoints or requests, even if those answers are factually incorrect @xcite . this behavior often results in misidentifying arguments, which are either nonexistent or irrelevant to the target event. output contradiction trap: when llms are forced to generate content that conflicts with their intentions, the resulting inconsistencies can disrupt the prediction probability, significantly degrading their performance on eae tasks. based on the above observations, we analyze root causes and utilize countermeasures for the three preference traps, which are simple yet effective: prior knowledge trap: the reason for falling into this trap is that prior knowledge provided fails to cover all actual annotations or that llms overfocus on specific patterns. the prior knowledge discussed here includes the definitions of event types and argument roles. first, we refine these definitions to better reflect actual meanings. second, we adjust the expressions of these definitions to avoid the llm becoming overly fixated on certain details. ## method as illustrated in figure 2 , the cat framework divides the eae task into the think stage and the choose stage, with countermeasures integrated (section 2.1). to address the prior knowledge trap, cat refines the definitions and revises biased expressions therein (section 2.2.1). to mitigate the sycophancy hallucination trap, cat provides clear output specifications and restricts the extraction scope (section 2.2.2). to tackle the output contradiction trap, cat adjusts the prompts and hard constraints to guide the llm output in its preferred format and natural habit (section 2.2.3). the specific prompts are presented in appendix a. ## results and analysis to evaluate the performance of our cat framework, we compare it with several strong baselines (section 4.1). then, we investigate the three preference traps in llms of different families and scales (section 4.2). the ablation study demonstrates the effectiveness of each component in cat (section 4.3). finally, we evaluate cat's model adaptability (section 4.4) and discuss the time cost (appendix c). ## conclusion in this work, we first experimentally investigate and summarize the preference traps in llm-based unsupervised eae tasks. we then propose cat, a two-stage framework employing practical countermeasures and an innovative generation approach for llms. experiments demonstrate that cat (using a local 7b model in a zero-shot setting) matches the performance of the best deepseek-r1. additionally, cat exhibits low time cost and robust model adaptation capabilities in further experiments. future work will extend this work to information extraction and control extraction boundary.