Word-Label Alignment for Event Detection: A New Perspective via Optimal Transport (2022)

Abstract: Event Detection (ED) aims to identify mentions/triggers of real world events in text. In the literature, this task is modeled as a sequence-labeling or word-prediction problem. In this work, we present a novel formulation in which ED is modeled as a word-label alignment task. In particular, given the words in a sentence and possible event types, the objective is to infer an alignment matrix in which event trigger words are aligned with the most likely event types. Moreover, we show that this new perspective facilitates the incorporation of word-label alignment biases to improve alignment matrix for ED. Novel alignment biases and Optimal Transport are introduced to solve our alignment problem for ED. We conduct experiments on a benchmark dataset to demonstrate the effectiveness of the proposed model for ED.

## introduction event detection (ed) is one of the critical tasks in information extraction. its goal is to identify and classify event triggers, i.e., the words/phrases that most clearly refer to the occurrence of an event of some predefined types in text. for example, in the sentence "joe biden was born on november 20, 1942", an ed system should recognize the word "born" as a trigger word of an event of type birth. a major challenge for ed is to assign an appropriate event type label for each word in a given sentence. in this work, we introduce a new perspective to solve ed as a word-label alignment problem that aims to align the set of words in the input sentence with the set of possible event type labels to represent correct label assignment for words. a key requirement for ed models in this new perspective involve inferring an alignment matrix to capture an alignment likelihood score between each pair of words and label types. the models can then be trained by enforcing the similarity between the predicted alignment matrix and the golden alignment matrix (computed from training data). in this way, previous ed models can be seen as a way to achieve the alignment matrix between words and labels where label distributions computed by the models serve as the alignment likelihood scores @xcite @xcite @xcite . however, given the word-label alignment perspective, previous ed models are suboptimal in at least two ways. first, the alignment likelihood scores in prior models are only used locally for each word (i.e., to compute the cross-entropy loss for each word to train models). the global uses of alignment matrix (e.g., to compute an overall distance between words and labels for training signals) are thus not yet explored in previous ed models. second, current ed models mainly obtain alignment likelihood scores based on representation vectors for words and types, thus unable to exploit assignment biases to improve quality of the alignment matrix to train ed models. in particular, we propose two types of alignment biases that can be helpful for ed: (1) word preference: words with high likelihoods to be event triggers should be more aligned with event type labels (i.e., not the other type for non-trigger words), and (2) type preference: event types that have higher chance to be appear in the input sentence should be associated with greater alignment scores. in all, we expect that global application and alignment biases can provide complementary information to boost current ed models in the new perspective. to implement this idea, we propose to encode event trigger likelihoods for words and appearance likelihoods for event types as two distributions over words and event type labels (respectively) that will be induced from a deep learning architecture. next, to inject the alignment biases into our ed model, we propose to feed the two distributions into optimal transport (ot) @xcite to induce an alignment matrix between words and event type labels. ot is an established framework to find the optimal alignment between two distributions, thus providing a decent solution to incorporate alignment biases to compute alignment matrix in our ed problem. finally, the induced alignment matrix will be leveraged to obtain a distance between words and event type labels, serving as a global application of the alignment matrix to introduce new training signals for ed. we conduct extensive experiments on a benchmark dataset to deliver state-of-the-art performance for ed. in summary, our contributions include: ## model given an input sentence @xmath0, w 2 , . . . , w n ], the goal of ed is to predict the label sequence @xmath1, l 2 , . . . , l n ] where l @xmath2. here, the label set t involves the bio encoding tags for the event types in a given event ontology (e.g., b_birth, i_birth, and other). in this work, we propose to model ed as a wordlabel alignment problem where an alignment matrix is formed to capture the assignment likelihood for every pair of words in s and labels in t . we will first discuss word/label representations, and alignment matrix computation for training afterward. word & label representation: to represent the words in s, following prior work @xcite , we employ the pre-trained bert model @xcite . concretely, the input sentence [[cls], w 1 , w 2 , . . . , w n ] is fed into bert to compute the contextualized embedding vectors @xmath3, e 1 , e 2 , . . . , e n ]. we employ the average of vectors in the last layer of bert to produce e. for the words with multiple word-pieces, we take the average of their word-piece representations. to represent the event type labels l i , we employ a randomly initialized embedding table t in which every label is represented by a vector t i . the representations of the labels are updated during training. alignment: to predict the label sequence l with our alignment idea, for every word w i , an alignment likelihood score a i,j between w i and each label l j is required (i.e., forming an alignment matrix a). using the scores a i,j , the label li can be predicted by @xmath4,j . note that in prior ed models, the alignment scores a i,j are directly computed using the final task-specific feed-forward networks @xcite . this approach is equivalent to computing the similarity between the representation vectors w i and t j , e.g., via dot-product. we call this approach "vanilla alignment". however, as discussed in the introduction, vanilla alignment scores a i,j are solely dependent on the learned representations e i and t j . as such, they cannot incorporate the alignment biases into the alignment matrix for ed. to this end, we introduce two alignment biases that can be exploited to improve the word-label alignment for ed. in particular, for an effective ed model, we expect the words that are more likely to be event triggers to have higher alignment scores with event types. in contrast, the other words should be better aligned with the special label other. i.e., non-trigger. we call this bias "word preference" for ed. in addition, among all event types, it is expected that the event types that have higher chance to be mentioned in the input sentence to be associated with greater scores in the alignment matrix a. we name this bias as "type preference". in this work, we aim to modify the vanilla alignment approach such that the two aforementioned preferences are observed. the quantification of word and type preference and their incorporation into alignment matrix will be discussed in the following. word & type preference: to compute the word preference and type preference in the input sentence s, we consider two simpler versions of the ed problem. specifically, for word preference, we utilize the trigger identification (ti) task that seeks to recognize the event trigger words without classifying them by event types. the event trigger probability computed for ti can be used to quantify the event trigger likelihood for each word w @xmath5. concretely, the representation e i of w i is fed into a feed-forward network with sigmoid activation function to compute the trigger likelihood p w i for w i : p w @xmath6, where σ and f f w are sigmoid and feed-forward layer, respectively. to supervise the trigger likelihood scores, we include the binary cross-entropy loss function for ti into the overall loss for training: next, for the type preference, we exploit the task of type prediction (tp) for ed. in this task, the objective is to predict which event types are mentioned in the sentence s (i.e., without predicting the trigger words). for an event type label t j , we predict the likelihood for t j to be mentioned in s by concatenating the type representation t j with the sentence representation e cls and feeding the result into a separate feed-forward network f f t with sigmoid activation to obtain the appearance likelihood for t j : p t @xmath7, e cls ])). to supervise the appearance likelihoods, the binary crossentropy loss function for tp is employed: formally, given the probability distributions p(x) and q(y) over the domains x and y, and the cost/distance function c(x, y) : x × @xmath8, ot finds the optimal joint alignment/distribution π * (x, y) with marginals p(x) and q(y) that converts p(x) to q(y) (i.e., the cheapest plan), by solving the following problem: here, π(x, y) involves all joint distributions with marginals p(x) and q(y). as such, the joint distribution π * (x, y) is a matrix whose entry (x, y) (@xmath9, @xmath10. we use the sinkhorn algorithm to approximately solve ot @xcite . finally, given π * (x, y), one approach to employ its global information is to compute the cost of optimal conversion dist(π * ) = σ @xmath11, y)c(x, y) to measure the distance between x and y (i.e., the wasserstein distance). to apply ot in our model, the domains x and y are defined as the words w @xmath12; the distributions p(x) and q(y) are set to the preference distributions d w p and d t p ; and the cost function c(w i , t j ) is computed using the euclidean distance between the representations e i and t j . as such, solving the ot equation leads to the optimal alignment π * (w i , t j ), serving as our predicted alignment matrix (i.e., a i,@xmath13, t j )). to train the ed model with word-label alignment, we propose two training signals obtained from the predicted alignment π * (e i , t j ). first, by treating the alignment score π * (e i , t j ) as the probability for w i to be assigned with label t j , we employ the negative log-likelihood loss to train our model: as the baselines, we compare with the typical sequence labeling models for ed, i.e., bilstm, dm-bert (bert with dynamic multi-pooling), and bert+crf in @xcite , and the prior state-of-the-art (sota) model reported for ace 2005, i.e., @xcite . for all the models, we use the same version of pre-trained bert base to achieve a fair comparison. following prior work @xcite , we use span-based precision, recall and f1 scores for correctly predicting the boundaries and types of event triggers as the performance metrics. finally, we fine-tune the hyper-parameters for oted using the development data of ace 2005. in our model we use the bert base model to encode data; 2 layers for all the feed-forward neural networks with 200 hidden dimensions in the layers. the trade-off parameters α task , α ot , α t i and α t p are set to 1.0, 0.01, 0.05, and 0.01 respectively. the learning rate is set to 3e-5 for the adam optimizer and the batch size of 8 is employed during training. results: the model performance is presented in ## related work early methods for ed employed feature engineering models @xcite @xcite @xcite @xcite . recently, deep learning was adopted as the sota approach for ed @xcite @xcite @xcite @xcite @xcite . unlike such prior work, we introduce a new word-label alignment perspective using ot for ed. finally, some recent work has utilized ot for character/word/example alignment problems @xcite @xcite @xcite . however, none of them explores ot for word-label alignment in ed. ## conclusion we present a general word-label alignment formulation for ed in which each pair of words and types is associated with an alignment score for label assignment likelihood. moreover, we introduce two alignment biases based on type and word preference to improve the word-label alignment matrix computation with ot. extensive analysis on a benchmark dataset demonstrates the benefits of the proposed technique for ed. in the future, we plan to evaluate our method on more datasets for ed @xcite @xcite to better understand its operation.