Pairwise Representation Learning for Event Coreference (2022)

Abstract: Natural Language Processing tasks such as resolving the coreference of events require understanding the relations between two text snippets. These tasks are typically formulated as (binary) classification problems over independently induced representations of the text snippets. In this work, we develop a Pairwise Representation Learning (PairwiseRL) scheme for the event mention pairs, in which we jointly encode a pair of text snippets so that the representation of each mention in the pair is induced in the context of the other one. Furthermore, our representation supports a finer, structured representation of the text snippet to facilitate encoding events and their arguments. We show that PairwiseRL, despite its simplicity, outperforms the prior state-of-the-art event coreference systems on both cross-document and within-document event coreference benchmarks. We also conduct in-depth analysis in terms of the improvement and the limitation of pairwise representation so as to provide insights for future work.

## introduction in this work, we study the event coreference resolution problem. event coreference resolution is commonly modeled as a binary classification problem over independently induced representations on the text snippets of each event mention @xcite . 2 understanding the relations between two text snippets is the essential part in the tasks. in this work, we argue that the representations of prior work are not expressive enough to learn the pairwise relations due to the following two reasons: (i) counterpart unawareness. the relationship between two mentions can be different in different contexts. to address different scenarios, it is better for each mention to ensure that its representation is aware of what its counterpart's representation. however, most early work induces mention representations independently by extracting features only from the sentence that contains the mention, without using the context of the other mention @xcite . some more recent work attempts to encode the whole document to represent each mention @xcite . this is beneficial for short documents, since the representation of each mention will also include information from the context of the other candidate mention. however, this is not sufficient for cross-document settings, when the comparison is, for example, between two event mentions that appear in separate documents. in this case even encoding large pieces of text leave the candidate mention representations independent of each other. (ii) unstructured representation learning. an event mention consists of multiple arguments that describe the event: who, when, where, etc. when determining the relationship of two event mentions, the mismatch of some arguments could be decisive. consider the following two sentences s 1 and s 2 (event trigger is underlined; argument #0 is in blue, location is in purple) s1: "over 69,000 people lost their lives in the quake, including 68,636 in sichuan." s2: "up to 6,434 people lost their lives in kobe earthquake and about 4,600 of them were from kobe." these two events "lost" are not the same events because the earthquake in sichuan and the earthquake in kobe are two different earthquakes, and sichuan and kobe do not have any geographic overlap. the mismatch of the locations "sichuan" and "kobe" may be enough to determine that the two events are different from each other without even considering the rest of the sentence. most prior work encodes all of the arguments into a single distributed representation vector and just compares the overall vector representations of two mention triggers. although contextual representation could encode all of the arguments' information, this is less optimal than explicitly representing all of the arguments, thus making it easier for the model to conduct fine-grained reasoning over each of the argument. to address the drawbacks of prior representations, we propose pairwise representation learning (pairwiserl). pairwiserl alleviates the aforementioned two limitations with two designs: pairwise representation learning. we suggest treating a mention pair, rather than a single mention, as the object for the representation learning. we encode the two mentions' sentences as a whole sequence so that one sentence's token representation is able to interact with the other sentence's from the very beginning. this is advantageous over learning two separate and independent representations because it allows for learning how compatible one mention is with the other mention's context. ## related work in this section, we discuss prior representation learning approaches for event coreference and how pairwise representation learning has been used in other nlp problems. event coreference. earlier work uses handengineered event features to represent events @xcite . most recent neural models use contextual embedding and character-based embedding of event triggers with some pairwise features to represent events @xcite @xcite . these works do not use argument information, and expect the contextual embedding includes all the necessary information. argument information has been integrated into event representations either by encoding some string-level features @xcite or by entity-level coreference co-training @xcite . in contrast, our representation learning of events has a unified system to encode the event triggers and the argument entities, which avoids the costly co-training while learning more advanced features that express the arguments on their own and their interactions with the event triggers. pairwise representation learning in other nlp tasks. pairwise representation learning has been widely adopted to model the relationships of two pieces of text. the main goal is to learn contextualized sentence representations. earlier systems commonly implement with attention mechanisms in recurrent @xcite , convolutional @xcite or transformer-style @xcite neural networks to deal with text generation, such as neural machine translation @xcite , document reconstruction @xcite , and document summarization @xcite ; machine comprehension @xcite , textual entailment @xcite , etc. in this work, we develop the pairwise representation learning for modeling the relationship of two mentions within two separate sentences rather than the relationship of the two sentences themselves. to the best of our knowledge, we are the first to (i) study pairwise representation for event pairs by letting two mentions learn from each other's context from the beginning 3 , and (ii) build structured representation between events by fine-grained argument reasoning, without any hand-engineered features. ## pairwiserl for coreference pairwiserl takes two sentences containing each mention as the input and outputs a score indicating how likely the two mentions refer to the same event. given the mention pair e i and e j with their arguments [arg0; arg1; loc; time], as shown in fig 1 , we concatenate the sentences of e i and e j , and encode the concatenated sentence using roberta @xcite . after encoding each token of the sequence to a representation vector, we sum up the token representations of the mention span as the representations for event trigger and event arguments respectively: v t for event trigger, v arg0 /v arg1 for argument #0 or #1, v loc for location and v time for time. next, we conduct fine-grained coreference reasoning, as figure 2 shows. the goal is to let each role of event arguments learn its contribution to the final task. for each role, where @xmath0, arg0, arg1, loc, time}, we first build the following rolewise representation: since entities do not have arguments, the final representation pairwiserl for entity coreference is: once obtaining the pairwise representation v(i, j), another four-layer mlp, as shown in figure 2 , will act as a binary classifier (i.e., is coreferential or not) ## experiments we apply pairwiserl to cross-document and within-document event coreference problems. ## analysis to further understand why pairwise representation performs much better than unpaired representation, and what limitations pairwise representation still has, we do a quantitative analysis on the errors of pairwiserl and the errors of the unpaired baseline model on ecb+. for each model, we randomly sample 100 errors: 50 false negatives and 50 false positives. false negative means that the gold label of the event pair is "coref", but the model predicts "not coref". false positives mean that the gold label of the event pair is "not coref", but the model predicts "coref". we manually classify these errors into different types, and study the difference between the error distributions of the two models. ## conclusion in this work, we propose a simple representation learning scheme, pairwiserl, for event coreference. pairwiserl learns a mention-pair representation by forwarding concatenated sentences into roberta, where sentences provide the context of mentions. this representation is applied to both within-document and cross-document event coreference benchmarks and obtains state-of-the-art performance. in addition, we augment this pairwise representation with structured argument features to further improve its performance.