Predicting Coreference in A bstract M eaning R epresentations (2020)

Abstract: This work addresses coreference resolution in Abstract Meaning Representation (AMR) graphs, a popular formalism for semantic parsing. We evaluate several current coreference resolution techniques on a recently published AMR coreference corpus, establishing baselines for future work. We also demonstrate that coreference resolution can improve the accuracy of a state-of-the-art semantic parser on this corpus.

## introduction abstract meaning representations (amrs, @xcite ) are a popular type of symbolic semantic representation for semantic parsing. amrs are labeled directed graphs whose nodes represent entities, events, properties, and states; the edges represent semantic relations between the nodes. for instance, in the example amrs of fig. 2 , the predicate node c describes a come-back relation between the arg1 "i" and the arg3 "this". amr is designed to abstract over the way in which a certain piece of meaning was expressed in language; thus "the destruction of the room by the boy" and "the boy destroyed the room" are represented by the same graph. in the example amr, the noun phrase "university offers" is decomposed into two nodes: the predicate node o:offer-01 and the argument node u:university, describing an event in which the university offers something to "i". an amr graphbank annotates each sentence in the corpus with an amr graph. recently, @xcite introduced the multi-sentence amr (ms-amr) corpus, which adds a layer of annotation on top of the amr-2017 graphbank that represents coreference and implicit arguments beyond the sentence level. an example is shown in fig. 1 . @xmath0; these mentions are not pieces of text as in other coreference annotation schemes, but nodes in the amr graphs. the annotation also specifies what implicit roles of predicate nodes the entity fills. in this paper, we make two contributions. first, we evaluate the performance of different coreference resolution tools on the ms-amr annotations. we evaluate these on the token level (by projecting the coreference annotations from the nodes to the sentences) and on the node level (by projecting the tools' coreference predictions to the nodes of the graphs) and find that allennlp with spanbert embeddings @xcite generally performs best. second, we show for the first time how the output of a coreference system can be integrated into the predictions of a state-of-the-art amr parser. we use the neural semantic parser of lindemann et al. ## coreference in ms-amr coreference resolution tools typically predict coreference between tokens in a text, but ms-amr annotates coreference between nodes in the amr graphs. to perform coreference resolution on ms-amr, we therefore have to map between the token level and the node level. the ms-amr corpus contains annotations which map between tokens and nodes, but this mapping is not always one-to-one. in the example shown in fig. 2 (a) , the two tokens "who knows" are aligned to the single node p. the nodes a3:amr-unknown and h:have-degree-91 are left unaligned. furthermore, amr graphs sometimes contain nodes that participate in the coreference chains but are not realized at the token level. for instance, in the sentence "speak to a doctor" the predicate speak-01 has an arg0 you which is a separate node in the graph even though it does not have any token alignment. we evaluate coreference tools on ms-amr in two different modes: token-level, where we project ms-amr coreference annotations from nodes to tokens and compare them against the predicted token-level coreference annotations; and node-level, where we project token-level coreference predictions to ms-amr nodes and compare them against the ms-amr annotations. because of the node-token mismatch explained above, we can project to the token level only coreference annotations between nodes that are aligned to tokens. we retained only coreference chains with at least two members. this reduces the 87 coreference chains between 425 mentions in the original ms-amr test set to 69 coreference chains between 385 mentions. 35% of these chains consist only of two mentions although there are also some very long chains with more than 30 elements, mostly pronouns. for the node-level evaluation and the smatch-based evaluation (see below), we used the unmodified coreference annotations on the nodes. ## comparative evaluation of coreference resolution tools we compared the output of the deterministic corenlp @xcite and neural corenlp @xcite coreference resolvers and tested two versions of the allennlp @xcite coreference tool based on the glove @xcite and spanbert @xcite embeddings respectively. these tools were chosen due to their availability and their strong accuracy on english. to evaluate the performance at the token level, the gold alignments were extracted and each coreference chain from the ms-amr dataset was mapped to the corresponding span in the text. these annotations represent the gold standard to which we compared the system annotations. in order to annotate coreference chains, a separate text file was created for each document with the sentences representing the document amrs. then each document text was processed with different coreference resolution systems to generate the predictions. for the token-level evaluation we compared the system output directly to the coreferent tokens in the ms-amr test set and for the node-level evaluation we first projected token annotations to the graph nodes using the gold alignments and then compared the node coreference chains. neural corenlp and allennlp are reasonable baselines for amr coreference resolution, although the results seem to be worse than state-of-the-art performance reported on news and narrative texts. one problem might be that the ms-amr corpus contains text snippets from blog data, including misspellings, jargon and incorrect grammar. also the conversational style used in blogs poses challenges for the coreference tools since they do not distinguish between posts made by different authors. the results of the node-level evaluation can be found in ## amr parsing with coreference coreference is not an isolated task in ms-amr parsing; in order to predict the gold annotations, coreference information needs to be incorporated into amr graphs predicted by a semantic parser. we thus first, we prepared gold annotations at the document level. for this, we combined the individual amrs from each document into a single graph to represent document-level annotations. the coreference chains were extracted from the gold annotations of the ms-amr corpus, and coreferent nodes in the document graph were merged following the procedure described in (o' @xcite . second, we ran lindemann's parser on each sentence separately and combined the predicted amr graphs into a document-level graph. then we ran spanbert allennlp (henceforth just allennlp) on each document text, and mapped each token-level prediction to the nodes the lindemann parser predicted for those tokens. we collapsed the coreferent nodes by replacing all edges into a node for a coreferent token by edges into the first node of the coreference chain; see @xcite for details. for example, in fig. 2 (a) there are three coreferent nodes i:i, i2:i and i3:i. since all three nodes represent the same entity the corresponding edges can be rearranged to point to the same node i:i as shown in fig. 2 (b) . we evaluated the performance of lindemann's parser, with and without the added coreference information, on the complete ms-amr test data. to this end, we computed the smatch score @xcite for the predicted vs. gold document-level graphs. it is worth noting that the overall smatch score is much lower than on other amr graphbanks; for instance, @xcite report a smatch f-score of 0.75 for their parser on the amr-2017 test set. even on the ms-amr test corpus without coreference links (i.e. pure sentence-by-sentence parsing), the parser only gets a score of 0.61, indicating that this is a harder corpus than amr-17. this then drops to 0.53 once nodes in the gold graphs are merged based on the coreference annotations. the coreference chains annotated in the ms-amr corpus are quite heterogeneous. at the token level, mentions of the same chain can be expressed as verbs, nouns or pronouns and fig. 3 illustrates one example where the chain includes different concepts at the node level: it, thing, harm-01, cut-01. such chains are hard to predict for the allennlp coreference model because they are realized as different parts of speech and are semantically nontrivial (harm/cut). 35% of all coreference chains in the test set are heterogeneous, i.e. they include entities that are expressed with multiple different parts of speech. ## discussion on the one hand, amr parsing already resolves some cases of coreference within the amr graphs. for instance, in fig. 2 (a) a single node o:offer-01 aligns to coreferent tokens "offers" and "them". on the other hand, some amr nodes can build coreference chains but do not have any token alignments. for example, a sentence like "speak to a doctor" has a separate node "you" as arg0 of "speak-01" in the amr graph. however, this node does not correspond to any token in the text. 9% of all coreferent mentions in the ms-amr test set do not have any alignments and the token-based coreference resolvers are not able to handle them. incorrect (or incomplete) node-token alignments can hurt the performance. 10% of all coreferent nodes in the test set refer to generic concepts like t:thing or p:person. this becomes a problem when allennlp finds the coreference with more specific nodes such as d:dad in fig. 4 . token "dad" is aligned to the node d:dad in the amr graph whereas the more generic node p:person does not have an alignment. however, the gold coreference chain includes only p:person as a member which results in the wrong classification of d:dad as false positive although both nodes actually correspond to the same entity. this example illustrates the problem when the gold annotation includes generic concepts that are represented in the amr graphs but not realized at the token level. we also found cases of incorrectly resolved personal pronouns because some texts were extracted from forums and the speaker could switch in the middle of the conversation, so that i and you would get a different meaning. for example, one document in the ms-amr test set contains the following text: "or should [i] 1 ... just keep an eye on the anxiety until it becomes a problem? well [i] 2 woudn't try to keep an eye on anxiety for a start because that will make [u] 1 tense." the first sentence has the pronoun @xcite 1 that refers to the same entity as [u] 1 in the second sentence and the [i] 2 pronoun in the second sentence corresponds to a different speaker. since the input text for the coreference tool does not include any meta information about the speakers the tool resolves both occurrences of "i" as referring to the same entity. this issue affects 9% of the coreference chains from the ms-amr test set. ## conclusion in this paper, we evaluated two popular coreference resolution tools on the ms-amr dataset, and found that the spanbert version of allennlp performs best in both a token-level and a node-level evaluation. we further extended a state-of-the-art amr parser with predicted coreference information, and obtained a three-point improvement in smatch score. the coreference models we have used here were quite conservative, in that they relied only on textual information. in the future, it would be interesting to extend them with features based on the amr graphs, which abstract over some surface details. it would also be interesting to predict bridging coreference relations and include those in the parser output too.