DEGREE ˆ2: Efficient Extraction of Multiple Events Using Language Models (2024)

Abstract: enter abstract here

## introduction in the domain of information extraction (ie), event extraction is a task consisting of identifying specific occurrences of things which happen involving participants @xcite . this task poses a number of unique challenges for information extraction systems, as proper detection of events typically requires an in-depth understanding of the semantics of input sentences, as opposed to simple lexical information. for example, the sentence "john went to san antonio" denotes a movement:transporttype event, whereas the sentence "the first point went to san antonio" does not. the bulk of the literature on event extraction descends from the original ace2005 information extraction dataset published by the linguistic data consortium @xcite . notably, this decomposes the event extraction task into two subtasks: event detection (also known as trigger extraction) john met with alice and then steve. [...template...] <@xmath0. john and alice met at some place. <@xmath1. john and steve met at some place. figure 1 : fine-tuning prompts used in our work compared to degree @xcite . text in blue denotes the input text to perform the event detection on. [...template...] represents the input template (section 3), with the following text being the expected generation of the large language model (llm). text in violet denotes the trigger phrase, teal the event participants, and magenta the event location. finally, orange text denotes special tokens added to the model vocabulary. at inference time, the llm generates text after the input source portion. and argument extraction. for example, in the sentence "john met with alice", "met" is the trigger (the phrase which clearly expresses the occurrence of the event), while "john" and "alice" are the arguments of the event. arguments can have a number of different event-specific types, such as meeting participants, locations, and relevant actors (e.g. the victim of a crime). supervised machine learning is a natural choice for modeling this problem, but the drawback of these approaches is that such training generally requires a large quantity of annotated data due to the need to understand the semantic nuances of text when performing this task. anecdotally, this can be prohibitive in a number of real-world ap-plications of event extraction systems, due to the fact that downstream users often (a) require a diverse set of event types and (b) these event types are many times unique to their use case (preventing useful sharing of annotated datasets between different users). with the advent of powerful language models and large language models (llms) @xcite , a number of novel lowresource and zero-shot methods have been developed which leverage these models' abilities to be fine-tuned to new tasks with relatively little data. one such model, known as degree @xcite , was until recently considered the state of the art in few-shot event extraction until being supplanted by the ee-lce @xcite model. while this would suggest a superior method for fine-tuning llms for event extraction, we find that this performance gap can be explained away by controlling for a specific limitation of the degree model: its inability to extract more than one event from the same region of text. in summary, our contributions are as follows: ## related work the bulk of research into event extraction focuses on high-resource scenarios, with models based on traditional supervised machine learning techniques. examples of this include techniques based on decision trees @xcite , support vector machines @xcite , convolutional neural networks @xcite , recurrent neural networks @xcite , and graph convolutional neural networks @xcite . broadly speaking, all of these approaches are based on the idea of training a machine learning algorithm from scratch to recognize event triggers and arguments using features which are either hand-crafted or, in the case of the neural network-based algorithms, automatically learned. more recent approaches to event extraction leverage language models. the basic idea of these techniques is to leverage the natural language modeling capacity of pretrained language models in order to reduce training data requirements via posing event extraction as a text-based natural language generation task. consequently, these techniques focus more on few-shot and zero-shot learning scenarios. the state-of-the-art in this space is ee-lce @xcite , which is an extension of instructuie @xcite . these flan-t5-xxl-based (chung et al., 2022) models are trained via a multitask learning algorithm designed to cover a large number of information extraction tasks. their results slightly beat out the previous state-of-the-art, known as degree @xcite , which is the inspiration for our work. for a more detailed history of event extraction datasets and systems, see @xcite . ## methodology before describing our extension to the model, we first provide a brief overview of the design of de-gree @xcite . the system frames the event extraction task in terms of a natural language generation task, with the generated text being rigidly structured in order to be machine-parsable. consider the sentence, "john met with alice." de-gree might query this input for contact:meet events with the following input: john met with alice. contact event, meet sub-type the event is related people meeting. similar triggers such as meet, met. the event's trigger word @xmath2. some people met at somewhere. the final two lines serve as a "prototype" template that should appear in the output. in this instance, we expect the fine-tuned model to produce the following completion: event trigger is met. john and alice met somewhere. for inputs where no event is found, the completion event trigger @xmath3. degree is trained by fine-tuning a base llm to complete patterns such as the above. once trained, the llm is able to extract not only the event types which it was trained on, but also, to some extent, new event types in a zero-shot fashion. these completions are easily parsable into structured formats, and the generated strings can be searched for in the original input in order to function as a text annotation algorithm. as shown above, degree is able to extract zero or one events from a given piece of text. how can entire documents then be handled? degree addresses this by chunking input documents into pieces consisting of three sentences foot_0 . all of a document's chunks are processed separately (once per event type) in order to perform event extraction across the full input. one remaining limitation is the handling of multiple events of the same type in the same chunk. degree does not address this situation, so we propose an update to the fine-tuning template structure which allows this type of scenario to be handled. our proposed template is shown in figure 1b . the key modification is the introduction of @xmath4, which separates each event in the output. while a rather minor change, we show below that this is enough to close the gap between degree and the state of the art. ## experimental results we evaluate our system on a variety of configurations using the ace2005 dataset @xcite 's english data. to determine sentence boundaries, we use the babel street analytics text analysis framework. our models are based on t5-large @xcite ), as we empirically found this to be a better choice than degree's base model of bart @xcite . for different numbers of sentences used to chunk apart input documents, we train two versions of each model: one with multievents turned off (i.e. the same algorithm as de-gree, with our base model and template, limited to a single event per chunk), and one with multiple events per input chunk. additional training details can be found in appendix a. additionally, we compare against three baselines: degree, instructuie @xcite , and ee-lce @xcite . for degree and instructuie, we use the models published by the authors. for ee-lce, we use the provided training code to create a model. to focus on the most pertinent subset of the dataset, we limit our analysis to the five event types with the highest support in the test data: conflict:attack, contact:meet, movement:transport, personnel:end-position, and transaction:transfer-ownership. finally, since we feel that it is more representative of performance on argument extraction, we opt to use a muc-style @xcite formula for calculating f1. this is identical to the traditional formula, except partial matches are counted as 50% correct (rather than completely incorrect). when interpreting the data in figure 2 , we find that extending degree to support multiple events causes two changes in the behavior of the model. first, the event detection performance becomes very similar to the state-of-the-art ee-lce system, despite being based on a model with 750mm pa-rameters (in contrast to ee-lce's 11b). second, model is able to process more sentences at once without sacrificing accuracy. because degree requires num_chunks×num_event_types invocations in order to process a document broken apart into num_chunks pieces, this means that we can effectively halve (or more) the number of model invocations required to process a document. ## relaxed f1: co-arity-invariant comparison of event extraction algorithms our exploration into the impact of this single-event limitation of degree on its comparative performance led us to consider whether there was a way to compare the qualitative performance of these algorithms in a mathematical way. for example, suppose that there is a dataset where each sentence contains one conflict:attack event, and we run algorithms a and b on chunks of two sentences. algorithm a is limited to zero or one outputs per sentence, but it detects an attack event in each pair of sentences. in contrast, algorithm b can output an arbitrary number of events, and it detects both attack events in each pair. which is better? in an absolute sense, algorithm b outperforms, since we calculate precision and recall metrics with respect to the number of events contained in the document. for certain applications, however, we may be more interested in knowing which of the two qualitatively performs better. in a certain sense, these algorithms are equivalent, since within the scope of its limitations, a and b both extract attack events as much as is possible. to address this shortcoming, we present a new metric for event detection, which we call relaxed f1 scores. the formula for this score is derived from the partial-match-aware muc formulae (chinchor and sundheim, 1993) and defined by the following formula for "relaxed" recall: r @xmath5.5 × partial)extra possibleimpossible in this equation, "correct" and "partial" denote the number of correctly-extracted, partiallyextracted (extractions of the correct type but only a partial overlap with the correct location) events or arguments, "possible" the number of events or arguments in the gold annotation. "impossible" denotes the number of events greater than one in each chunk (i.e. for a chunk with five events, "impossible" would be four). finally, the "extra" term is needed for algorithms which can extract multiple events, in order to make the result comparable with ones which cannot. for these algorithms, "extra" denotes the number of correct (or weighted partial) predictions which were made that would have been impossible if multiple events could not be extracted. in sum, this means that, effectively, for each chunk of text produced during processing, the calculation of relaxed recall becomes binary. from this relaxed recall value, relaxed f1 is computed by using the standard formula alongside the standard precision p : we use this metric to determine whether our multiple event extraction extension qualitatively decreases the event detection performance of de-gree, with the results shown in figure 3 . this graph shoes a roughly linear correlation between the two values, meaning that our extension does not meaningfully degrade degree's qualitative performance. ## discussion we demonstrate that a simple extension to de-gree is sufficient to close the gap between it and state-of-the-art systems. this suggests that different generative approaches to event extraction are potentially much more competitive with one another than previously thought. furthermore, we present an f1-style event detection metric which can give some insight into the qualitative performance of these algorithms. we hope that this motivates further research into ways of analyzing these systems' performance in more fine-grained detail. future work could include a metric that allows for assessing argument extraction performance without depending on event detection accuracy. ## limitations the systems described in this paper are trained on annotated event datasets. while they have some capacity to generalize to new event types in a zeroshot fashion, users should be cautious when using them with event types not found in the training data, as they may produce unexpected predictions. the analysis presented here focuses on the english-language ace2005 data. some of the conclusions presented here may not hold for certain other languages, and the systems described here may not function correctly on non-english input text.