S peech EE @ XLLM 25: End-to-End Structured Event Extraction from Speech (2025)

Abstract: Event extraction from text is a complex taskthat involves the identification of event triggersand their supporting arguments. Whenapplied to speech, this task becomes evenmore challenging due to the continuous natureof audio signals and the need for robustAutomatic Speech Recognition (ASR). Thispaper proposes an approach that integratesASR with event extraction by utilizing theWhisper model for speech recognition and aText2Event2 Transformer for extracting eventsfrom English audio samples. The Whispermodel is used to generate transcripts from audio,which are then fed into the Text2Event2Transformer to identify event triggers and theirarguments. This approach combines two difficulttasks into one, streamlining the processof extracting structured event information directlyfrom audio. Our approach leverages arobust ASR system (Whisper) followed by aparameter-efficient transformer (Text2Event2fine-tuned via LoRA) to extract structuredevents from raw speech. Unlike prior worktrained on gold textual input, our pipeline istrained end-to-end on noisy ASR outputs. Despitesignificant resource constraints and datanoise, our system ranked first in the ACL 2025XLLM Shared Task II.

## introduction event extraction from speech audio samples poses a challenge as the shortcomings of asr like noise, substitution errors, hallucinations and other errors get propagated to the event extraction transformer leading to erroneous training dataset leading to the transformer learning from an erroneous data. unlike previous work that utilizes clean, curated textual data for event extraction, we address the more realistic and challenging scenario of extracting structured events directly from raw audio input @xcite . this introduces transcription noise, alignment challenges, and limited supervi-sion, requiring novel techniques to ensure generalizability and robustness. we have streamlined the process of event extraction from english audio samples with wise (whiper-to-structured-events) which utilizes whisper-medium foot_0 developed by openai foot_1 and fine-tuned text2event2 transformer model as in @xcite . the audio before passing through the whisper-medium model and tokenizer is processed to convert to a frequency of 16khz and monophonic channel audio samples. it is done to bring the audio samples to the same reference frame for better transcript generation which will lead to better event extraction and can also use multimodal llms like @xcite . loudness of the audio samples is also standardized to bring them to the same reference frame. then the audio samples are passed through the whisper-medium model and transcripts are generated. this transcripts of train and development dataset in combination with the labelled events of the audio files was used to fine-tune the text2event2 transformer model for better accustomed with our scenario. ## dataset description dataset was provided to us for a shared task organised by xllm in collaboration with acl in 2025. this dataset is specifically from the shared task ii: speech event extraction (speechee). the dataset contains 33 event types and 22 argument roles, with 19217 training data, 901 validation data and 676 testing data. the data was given to us inform of english audio samples. in addition to this, we were also given a detailed event schema in the form of json which included all the event types and the argument types to support a particular event. transcript: even as the secretary of homeland security was putting his people on high alert last month, a 30-foot cuban patrol boat with four heavily armed men landed on american shores. underly undetected by the coast guard secretary ridge now leads. { "trigger": "landed", "type": "transport", "arguments": [ {"name": "boat", "role": "vehicle"}, {"name": "men", "role": "artifact"}, {"name": "shores", "role": "destination"} ] } figure 2 : an example of events and their arguments the training set and development set was combined and created into one dataset since the labeled dataset was so limited. a small set is kept aside for testing. it is important to note that although the dataset mirrors the ace05en schema, no gold transcripts were provided. all training data was supplied as raw english audio, requiring the construction of training data via asr-generated transcripts. this modality shift introduces significant transcription noise, necessitating event extraction models that are robust to imperfect input. ## methodology automatic speech recognition also known as asr is used to convert human speech to readable text. it has grown quite recently and is being used in various fields where human speech need fast transcriptions like live caption generation and live translation from one language to another language. this all requires speech recognition and speech-to-text conversion models. whisper by openai @xcite is a state-of-the-art asr model trained on 6, 80, 000 hours of multilingual and multitasked supervised data. training on this vast dataset has made the model robust to background noise, accents, and various languages. we have used the whisper-medium model to generate the transcripts of the training set and the development set and created into a pandas data frame. file name transcription train-10589.wav oh, uh-huh. train-18281.wav and now just so... train-6191.wav at the time... train-140.wav and the democrats... train-12985.wav tom racings train-11948.wav i don't know. train-2803.wav it would talk about tips.... train-463.wav i did not feel less than train-2041.wav they got to understand. train-2815.wav famed world war ii... ## results and discussion to assess the effectiveness of event extraction models, organisers adopted a multi-task evaluation framework comprising three subtasks. each task evaluates different aspects of event structure and prediction quality. the evaluation metric for each task is the f1-score, computed from precision and recall. the final score is a weighted combination of the three task-specific f1-scores using the formula: this weighting reflects the relative importance of each task in capturing comprehensive event understanding. ## limitations a major limitation in our pipeline stems from the use of asr-generated transcripts without access to gold textual input. whisper, while state-of-theart, may hallucinate or omit important information, which gets propagated into the event extraction phase. furthermore, due to the exhaustion of gpu quotas on kaggle, the majority of training was conducted on cpu using lora, which limited the number of training epochs and speed of experimentation. for us, resource constraint has also been a major problem. we could only fine-tune the event extraction models for 3 epochs and limited gpu usage leading to large amount of training and testing time. ## future work event-tagged data can be expanded through humanannotated efforts, albeit at a significant cost in terms of time and labor @xcite . alternatively, data augmentation techniques can be employed to enhance dataset size and diversity. one effective method involves replacing event-triggering words and their corresponding arguments with appropriate synonyms using tools such as spacy @xcite or wordnet @xcite . moreover, while existing datasets like conll-2003 @xcite are comprehensive in terms of named entity recognition, their event schemas remain relatively limited. once the challenge of insufficient annotated data is addressed, alternative architectures beyond transformer-based models-such as bi-directional lstms @xcite and graph neural networks @xcite with attention mechanisms-can be explored. these models are capable of capturing deeper semantic relationships, thereby improving the performance of event extraction systems, as demonstrated in @xcite ) @xcite . ## conclusion this work demonstrates that effective event extraction from speech is possible even under compute-constrained, noisy-input scenarios. through the use of peft via lora and a robust asr+transformer pipeline, our system outperformed all other submissions in the xllm shared task ii. future work will explore improving robustness to asr noise and enhancing lowresource adaptability via data augmentation and semi-supervised learning.