S peech EE @ XLLM 25: Retrieval-Enhanced Few-Shot Prompting for Speech Event Extraction (2025)

Abstract: Speech Event Extraction (SpeechEE) is a challenging task that lies at the intersection of Automatic Speech Recognition (ASR) and Natural Language Processing (NLP), requiring the identification of structured event information from spoken language. In this work, we present a modular, pipeline-based SpeechEE framework that integrates high-performance ASR with semantic search-enhanced prompting of Large Language Models (LLMs). Our system first classifies speech segments likely to contain events using a hybrid filtering mechanism including rule-based, BERT-based, and LLM-based models. It then employs fewshot LLM prompting, dynamically enriched via semantic similarity retrieval, to identify event triggers and extract corresponding arguments. We evaluate the pipeline using multiple LLMs—Llama3-8B, GPT-4o-mini, and o1-mini—highlighting significant performance gains with o1-mini, which achieves 63.3% F1 on trigger classification and 27.8% F1 on argument classification, outperforming prior benchmarks. Our results demonstrate that pipeline approaches, when empowered by retrievalaugmented LLMs, can rival or exceed end-toend systems while maintaining interpretability and modularity. This work provides practical insights into LLM-driven event extraction and opens pathways for future hybrid models combining textual and acoustic features

## introduction information extraction aims at automatically identifying structured information, such as entities and their relations, from unstructured data @xcite . a task in this domain is event extraction (ee) @xcite searching for answers to questions like what happened, who was involved, and where did it take place. the source data can be text @xcite , but even images @xcite or videos @xcite . speech event extraction (speechee) @xcite extends textual ee, with the purpose of identifying structured event information directly from the input of the spoken language. this task occupies a unique intersection between automatic speech recognition (asr) and natural language processing (nlp), requiring not only accurate transcription but also the detection of event types, triggers, and arguments from possibly noisy spoken content. existing speechee approaches can be broadly categorized into the methodologies pipeline-based and end-to-end. pipeline-based architectures typically employ an asr module to transcribe speech, followed by text-based event extraction using nlp techniques @xcite . these systems offer modularity and transparency, allowing separate optimization and analysis of asr and extraction components. however, they are susceptible to cascading errors, where transcription inaccuracies can significantly impair downstream event extraction performance. conversely, end-toend approaches aim to bypass intermediate text by learning to map raw audio directly to structured outputs @xcite . while promising in reducing error propagation and potentially more efficient, these models often demand large-scale annotated audio-event datasets and are less interpretable, acting as opaque "black boxes" in many cases. recent progress in large language models (llms) such as gpt-4 (openai, 2024a) has opened new possibilities in pipeline-based architectures by enabling powerful few-shot and zero-shot learning capabilities. llms exhibit remarkable proficiency in extracting structured knowledge from unstructured text with minimal task-specific supervision @xcite . when combined with state-of-the-art asr systems, these models can form the backbone of robust speechee systems that generalize well across domains and require minimal adaptation. in this work, we present a pipeline-based speechee framework that leverages semantic search-enhanced few-shot prompting with llms. our system dynamically retrieves relevant examples from a support set and incorporates them into prompts to guide event extraction. additionally, we introduce a classification mechanism to identify utterances likely to contain events, reducing false positives and improving extraction precision. our main contributions are as follows: the remainder of the paper is organized as follows. section 2 reviews related work on speechee and language models. section 3 introduces the dataset. section 4 describes the model architecture and pipeline components. section 5 presents the experimental results and analysis. section 6 outlines the results of the ablation study. section 7 discusses key findings and limitations. finally, section 8 concludes the paper and suggests directions for future research. ## dataset we use the speechee shared task dataset 1 , derived from ace2005-en+. the dataset is provided in a structured json format, where each entry consists 1 https://xllms.github.io/speechee/ of a unique id, an event trigger indicating the lexical anchor of the event, its corresponding type, and a list of associated arguments, each annotated with a semantic role. in total, the dataset includes 33 distinct event types and 22 argument roles, offering a diverse and challenging benchmark. it comprises 19,217 training instances, 901 development examples, and 676 test samples. an example of the data format is shown below: {" id ": " train -6" , " event ": [ {" trigger ": " election ", " type ": " elect ", " arguments ": [ {" name ": " man ", " role ": " person "} ]} ]} 4 methodology our proposed pipeline for speech event extraction consists of several key components, as illustrated in figure 1 . the pipeline follows a modular approach, allowing for component-level evaluation and optimization. ## results we evaluated our event extraction pipeline using two primary metrics, following the evaluation protocol outlined by @xcite ): for both tasks, we report precision (p), recall (r), and f1-score (f1), with the f1-score serving as the primary measure of overall performance. ## ablation to substantiate our initial observation that llms tend to produce a considerable number of false positives, we conducted an ablation study evaluating all three models under various classification configurations. the results, presented in when applying a single criterion, each model demonstrated optimal performance with a different method: o1-mini performed best with the bertbased classifier, while 4o-mini and llama3-8b yielded higher scores with the rule-based and llmbased classifiers, respectively. although relying on the one+ filtering resulted in marginally lower performance than the best individual setups, it still outperformed the baseline without classification. notably, aggregating predictions with two+ and three led to consistent f1-score improvements over the standalone classifiers. in the ac task, the outcomes are more nuanced, indicating that strategies yielding gains in tc do not always translate to ac. for instance, while 4o-mini achieved the best tc performance with the rule-based method (when considering single criterion), it was outperformed by other approaches in ac. nonetheless, similarly to tc, both two+ and three yielded improvements over the individual classifiers. however, unlike in tc, the three criterion did not universally result in the best performance, benefitting only the o1-mini model. these consistent enhancements across both tasks confirm the effectiveness of the classification step in reducing false positives and improving overall model performance. ## discussion our experimental results demonstrate that a welldesigned pipeline approach can achieve performance comparable or even superior to state-of-theart end-to-end models for speech event extraction. this finding is significant as it challenges the assumption that end-to-end approaches necessarily outperform pipeline methods for complex tasks. our results suggest that by leveraging advanced llms and intelligent example selection strategies, pipeline approaches can mitigate traditional weaknesses such as error propagation. moreover, this approach offer several advantages over end-to-end models, including modularity and interpretability, in the sense that the output of each stage can be examined and is humanly interpretable. the minimal performance difference observed when comparing whisper and canary suggests that transcript quality variations within a certain threshold have limited impact on event extraction outcomes. both whisper and canary have their own characteristics, when it comes to the transcription of names or cities, which are crucial in event extraction. the fact that this did not make too much of a difference, is encouraging for real-world applications, where perfect transcription cannot be guaranteed. however, it is important to note that the performance gap might be more pronounced when comparing with lower-quality asr systems or when processing audio with significant noise, accents, or other challenging characteristics. the substantial performance differences observed between the three llms highlight the critical role of llm selection in event extraction efficacy. model size seemed to be a factor, with the larger proprietary models clearly outperforming the locally runnable llama model. the reasoning capability also largely seems to help the task, but to make these claims more confidently, a more thorough research would be needed across several models. during development, we also observed that in several cases, the false positives produced by the llms were remarkably close to genuine events that should have been annotated. this suggests that, in some instances, the models may have correctly identified events that were inadvertently missed during the original annotation process. ## conclusion this research presents an llm-driven pipeline for speech event extraction that achieves performance comparable to state-of-the-art end-to-end models. our approach combines asr-generated transcripts with semantic search-enhanced few-shot learning to create a modular and interpretable framework for identifying events and their arguments from spoken language. by dynamically selecting examples based on semantic similarity to the current input, our approach ensures that the llm receives the most relevant and informative context for each specific case. this is particularly important for event extraction, where different event types have distinct patterns, trigger words, and argument structures. the effectiveness of this approach suggests that similar techniques could be beneficial for other complex nlp tasks where pattern recognition plays a crucial role and where diverse examples exist in the training data. the choice of asr system showed limited impact on extraction performance, suggesting robustness to transcript quality variations within a reasonable range. however, llm selection plays a critical role in event extraction efficacy, with larger, more capable models achieving significantly better results. several promising directions for future research emerge from this work, including hybrid approaches exploring methods that integrate transcript-based and direct audio-based features that could potentially combine the strengths of pipeline and end-to-end approaches. our approach does not leverage speech-related cues present in the audio, which could potentially enhance performance if incorporated. with prompt engineering, the current prompting strategies could be further refined and may improve llm performance without requiring additional computational resources. another area could be exploring methods to reduce computational requirements, such as model distillation or selective component invocation. this work advances spoken language understanding by demonstrating that modular pipelines can rival end-to-end models through strategic integration of llms and retrieval mechanisms. by decoupling transcription from extraction while maintaining cross-component optimization potential, our framework offers a practical pathway for deploying speech event extraction tools.