Modeling Document-Level Context for Event Detection via Important Context Selection (2021)

Abstract: The task of Event Detection (ED) in Information Extraction aims to recognize and classify trigger words of events in text. The recent progress has featured advanced transformer-based language models (e.g., BERT) as a critical component in state-of-the-art models for ED. However, the length limit for input texts is a barrier for such ED models as they cannot encode long-range document-level context that has been shown to be beneficial for ED. To address this issue, we propose a novel method to model document-level context for ED that dynamically selects relevant sentences in the document for the event prediction of the target sentence. The target sentence will be then augmented with the selected sentences and consumed entirely by transformer-based language models for improved representation learning for ED. To this end, the REINFORCE algorithm is employed to train the relevant sentence selection for ED. Several information types are then introduced to form the reward function for the training process, including ED performance, sentence similarity, and discourse relations. Our extensive experiments on multiple benchmark datasets reveal the effectiveness of the proposed model, leading to new state-of-the-art performance.

## introduction event detection (ed) is one of the fundamental tasks for information extraction. its goal is to identify the word(s) in text that most clearly evoke events and classifying them into predefined event types (event trigger words). for instance, in the input sentence "after the scandal, david james was fired from the company.", an ed system needs to recognize the word "fired" as an event trigger and predict its event type as end-position. the early methods for ed have involved featurebased models @xcite @xcite while recent work has featured deep learning methods @xcite @xcite . similar to other nlp tasks, the current best systems for ed leverage transformer-based language models, e.g., bert @xcite , as a critical encoding component to achieve state-of-the-art performance @xcite . as such, most of the current transformer-based models for ed only focus on sentence-level context in which the scope of context to predict event type for each word is limited to the host sentence @xcite . however, it has been shown that document-level context also provides important information for deep learning models for ed @xcite . for instance, in the document "the troops were retreating cautiously. he was shocked after he heard "fire!".", to correctly predict attack as the type of the event evoked by "fire" (i.e., avoiding the confusion with the event type end-position), it is necessary to consider the previous sentence with the important context words of "troops" and "retreating". consequently, in this work, we aim to develop a transformer-based model that can effectively encode document context to boost the performance for ed. there have been a few recent work that applies bert-based models for document-level ed @xcite . however, as bert can only encode input texts with up to 512 sub-tokens (due to the quadratic self-attention complexity), current bert-based document-level models for ed has only constrained their applications to document context that can fit into the bert length limit. the typical approaches involve only considering short documents @xcite or truncating long documents @xcite , as also done for other tasks @xcite . these models are thus unable to encode long-range dependencies in document context that go beyond the length limit of bert to further improve the ed performance. to alleviate the length limit for bert-based ed models, two major approaches from other nlp tasks can be considered: (1) architecture change for self-attention @xcite @xcite : in this group, the vanilla self-attention of transformer is replaced with some variant mechanism such as sparse selfattention @xcite ) that can encode larger context with the same complexity as the original transformer (e.g., longformer @xcite can consume 4096 sub-tokens compared to 512 sub-tokens in bert); and (2) hierarchical design @xcite : these works employ the standard transformer-based language models to encode the context with certain length limit. for larger context, they combine the transformer model with other deep architectures to construct the final model for specific tasks. for instance, @xcite split the documents into multiple chunks (to fit the length limit of bert) and send the bert-based representations of the chunks into a recurrent neural network for document modeling. however, both solutions have their own drawbacks that might not be suitable for ed. for the former, the transformers with variants of vanilla self-attention are still limited to a certain length (e.g., 4096 for longformer), thus unable to encode important contexts that are farther away in the documents for ed. moreover, due to the changes in the self-attention mechanism, such variants generally lead to lower performance for nlp tasks compared to vanilla transformer @xcite . for the latter, standard transformer architectures are still employed to represent chunks of input texts to the extent of their length limits, thus failing to benefit from the ability of transformer to consume the entire input texts and capture long-range dependencies of documents for representation learning. in particular, the selfattention component of the models in this case is still bounded to the length limit and cannot directly capture the dependencies over the entire document. to this end, to develop an effective bertbased document-level model for ed, our motivation in this work is to explicitly select only important/relevant parts of a document for a given sentence and feed those context into bert for improved representation learning. one the one hand, context selection allows the model to compress an input document into a smaller chunk with important context that can fit entirely into the length limit of bert to better exploit the capacity of bert for representation learning. further, the ignorance of irrelevant context in the input for bert might also reduce noise and improve representation vectors for ed. in particular, given an input sentence s i in a document for event prediction, our method seeks to select a subset of sentences for s i in the document that contain the most important context for the event prediction of s i . the subset is also constrained so its concatenation with s i (i.e., the compressed document) can fall within the length limit of bert, thus enabling bert to encode the compressed document entirely. as such, the key question in our model is how to design the sentence selection to facilitate the encoding of important context in documents with bert for ed. to this end, we argue that the important context for an input sentence s i in a document should be determined by the improved performance of bert for ed on s i when s i is augmented with the context. as such, we propose to use the policygradient method reinforce @xcite to guide the context selection module using the performance of bert for ed as the reward. in addition, we introduce auxiliary rewards based on linguistic intuition (i.e., semantic and discourse relations between the input sentence s i and selected context sentences) to enhance the selection process. our extensive experiments on benchmark datasets show that the proposed method can achieve state-of-theart results for ed on both the sequence-labeling and the word-classification formulations. ## model in the literature, ed has been formulated as sequence-labeling @xcite or word-classification (nguyen and grishman, 2015) problems. in this work, we explore both formulations of the task. formally, given the input sentence s @xmath0, w 2 , . . . , w n ] (with n words) from the document @xmath1, s 2 , . . . , s n ] (with n sentences), the goal is to recognize and classify event triggers in s i , leveraging the broader context in d. in the sequence-labeling formulation @xcite , as event triggers are allowed to involve multiple words, ed aims to assign a label y t to each word w @xmath2, y 2 , . . . , y n ] can capture event trigger boundaries and types in s i . for the word-classification formulation @xcite , the model additionally has an index t for the trigger candidate word w @xmath3. here, we also include a special type none to indicate that a word is not an event trigger. as such, in the word-classification setting, the model performs separate event type predictions for each word in the input texts. as discussed in the introduction, our goal is to design a document-level model for ed that can effectively leverage the representation learning ability of bert (i.e., entirely encoding important context sentences for s i in ed using the powerful bert and self-attention). to this end, we propose to select a set c i that contains the most important context sentences for the event prediction of s i in d, i.e., c i ⊂ s @xmath4. the target sentence s i will be augmented with the sentences in c i to form a shorter document d (i.e., @xmath5. in our model, d will be constrained to fall within the length limit of bert, thus allowing bert to consume d entirely for improved representation learning for ed. in this way, the important context with arbitrary distance from s i in d can be reached and packed into d to allow effective document-context modeling with bert for ed. in the rest of this section, for brevity, we remove the subscript i from c i . our model for ed in this work consists of two major components: (i) prediction model: this model consumes the shorter document d and perform event type prediction for s i , and (ii) context selection: this component selects the important context sentence set c to perform ed for s i . ## experiments datasets & baselines: to study the effectiveness of the proposed model, called event detection with dynamic document context (ed3c), we evaluate its performance on two benchmark datasets ace 2005 @xcite and @xcite . we choose these two datasets as they provide documents that are much longer than the input length limit for bert base , thus being more suitable to our focus on document-context modeling for ed foot_1 . we use full document context for the documents in these datasets. ace 2005 annotates 599 documents for 33 event types. we use the same data split and preprocessing as prior work @xcite @xcite for this dataset. the numbers of documents for the training/test/validation data are 529/40/30 respectively. in prior work, the ed problem on ace 2005 has been addressed via both the sequence-labeling @xcite and word-classification @xcite formulations. the sequencelabeling formulation adheres to the original annotation in ace 2005 to allow multiple words in event triggers. the word-classification formulation, in contrast, simplifies the problem by only concerning the single most important words in event triggers @xcite . we use ace 2005 to evaluate the systems on both formulations in this work. as such, we compare the performance of ec3c with the current state-of-the-art (sota) ed models on ace 2005, i.e., bilstm @xcite (for sequence-labeling) and ekd @xcite cyseced is a recent ed dataset that annotates 300 articles for 30 cybersecurity event types. we use the same data split proposed in the original paper @xcite with 240/30/30 documents for training/test/validation data. following prior work, we evaluate ed models on the wordclassification setting. we compare ed3c with the current sota model on cyseced, i.e., deeb-rnn (word2vec) @xcite . in addition to the aforementioned baselines, we also compare the proposed model with prior stateof-the-art document-level models for ed, including hbtngma @xcite that employs gated multi-level attention mechanism and deeb-rnn (bert) @xcite that uses a bidirectional rnn for encoding the sentences of the documents. note that as bert is not originally used in these models, we use their provided implementations and inject the bert base model into the encoding components for a fairer comparison. furthermore, we compare ed3c with other variants of pre-trained transformer-based language models capable of encoding input texts with longer length than bert base limit. specifically, we consider three commonly used language models: bigbird @xcite which utilizes sparse selfattention, reformer @xcite which replaces dot-product attention with locality-sensitive hashing, and longformer @xcite that utilizes local self-attention together with global task-aware attention. finally, we compare ed3c with two typical solutions for document encoding using bert, i.e., hierarchical @xcite which splits a document into multiple chunks and encode them separately with bert, a bilstm model is then employed to aggregate the representations of the chunks; and neighbor sentences (schweter and akbik, 2020) which augments the current sentence with sentences immediately appearing before and after the current sentence in the document for bert-based encoding. note that for bigbird, reformer, longformer, and "neighbor sentences", we use both left and right context sentences for augmentation. the corresponding quota for input length of a model (e.g., l i for bert) is divided evenly for the two types of context. hyperparameters: we tune the hyperparameters for the proposed model using a random search. all the hyperparameters are selected based on the f1 scores on the development set of the ace 2005 dataset (with sequence-labeling setting). the same hyper-parameters from this fine-tuning are then applied for cyseced dataset and word-classification setting for consistency. in our model we use the bert base to encode data; 300 dimensions for the hidden states of lstm and 2 layers for feedforward neural networks with 200 hidden dimensions. the trade-off parameters α, β and γ are set to 0.5, 0.1, 0.05, respectively. the learning rate is set to 3e-5 for the adam optimizer and the batch size of 16 are employed during training. comparison: the performance of the models are shown in ## analysis context selection: to demonstrate the benefit of training the context sentence selection, we examine typical heuristics for choosing context sentences in a document d for the target sentence s i in ed. as such, given the target sentence, these heuristics can directly suggest a set of context sentences (i.e., no training). the prediction model m ed is then trained on the augmented texts of the suggested context and the target sentence for ed. note that in these heuristics, context sentences are selected until the bert base length limit is exceeded; the selected context sentences and target sentence are organized using their order in d to form the augmented short document d for bert. in particular, we explore the following selection heuristics in this section: (1) only-right and only-left: these approaches only augment the target sentence s i with its right context sentences (i.e., s @xmath7.e., s @xmath8; (2) neighbor sentences: this approach is described in the baselines for ed3c above. it uses both left and right context sentences to augment s i where the remaining space for context sentences in bert length limit (i.e., l @xmath9; (3) most similar: in this baseline, we select the top context sentences s j d that have the highest similarity with the target s i . as such, we first obtain a representation vector for each sentence s j by feeding s j to the pre-trained bert base model and using the hidden vector for the [cls] token in the last layer for this purpose. afterward, the similarity between s j and s j is computed via the cosine similarity of their representation vectors; and (4) highest coreference: finally, the selection strategy in this approach ranks context sentences s @xmath10, s i ), the normalized number of entities mentioned in both s j and s i (i.e., used to compute the reward r disc i . the context sentences with the highest cor(s j , s i ) scores are chosen for augmenting s i . the results of this analysis are provided in case study: to better understand the operation of ed3c, we manually analyze the examples in the development set of cyseced whose event types are not correctly predicted by the baselines (bigbird, reformer, longformer, and "neighbor sentence"), but can be successfully recognized by ed3c. two examples for this category is presented in ## related work ed has been approached with feature-based models earlier @xcite @xcite @xcite . recently, deep learning (dl) methods are proved to be an effective approach for ed @xcite @xcite @xcite @xcite @xcite . transformer-based language models such as bert @xcite are the core components for current sota deep learning models for ed @xcite . recently, there have been growing interests to solve ed in the low-shot learning settings to improve the data efficiency of the models @xcite . the majority of prior dl models for ed are restricted to sentence-level context. in recent years, encoding document context with dl has also been shown to be helpful for ed @xcite @xcite . however, existing dl methods for document-level ed have not employed transformer-based models, or only utilized them a limited manner (i.e., focusing on short documents or truncating long documents to fit the bert length limit). as described in the introduction, two main approaches have been explored to address the length limitation for transformer-based models: (1) transformers with modified self-attention @xcite @xcite , and (2) truncating or hierarchical modeling @xcite @xcite . the major drawback of these approaches involves the inability to capture important context of arbitrarily distance from the target and the failure to entirely consume important context with bert and self-attention for effective representation learning. these issues are address by ed3c to boost the performance for ed. ## conclusion we introduce a novel method, i.e., ed3c, to model document context for ed with bert. our model seeks to group important context sentences for the target sentence into a short document that can be entirely encoded by bert to produce effective representation vectors for ed. as such, ed3c presents a context selection component that sequentially selects relevant sentences in the document based on lstm. we use reinforce to train the context selection component in ed3c. our extensive experiments demonstrate the effectiveness of ed3c with sota performance on benchmark datasets for ed. in future, we plan to apply the proposed model on other related tasks (e.g., event extraction).