D oc AMR : Multi-Sentence AMR Representation and Evaluation (2022)

Abstract: Despite extensive research on parsing of English sentences into Abstract Meaning Representation (AMR) graphs, which are compared to gold graphs via the Smatch metric, full-document parsing into a unified graph representation lacks well-defined representation and evaluation. Taking advantage of a super-sentential level of coreference annotation from previous work, we introduce a simple algorithm for deriving a unified graph representation, avoiding the pitfalls of information loss from over-merging and lack of coherence from under merging. Next, we describe improvements to the Smatch metric to make it tractable for comparing document-level graphs and use it to re-evaluate the best published document-level AMR parser. We also present a pipeline approach combining the top-performing AMR parser and coreference resolution systems, providing a strong baseline for future research.

## introduction abstract meaning representation (amr) is a formalism that represents the meaning of a text in the form of a directed graph, where nodes represent concepts and edges are labeled with relations @xcite . until recently, the annotated corpora for amr only provided sentence-level annotations. as a result, amr parsing research has been limited to sentence-level parsing. a unified document-level amr graph representation can be quite useful for applications that require documentlevel understanding, such as question answering and summarization. the most recent release of amr annotations (amr 3.0) includes a multi-sentence amr corpus (ms-amr; @xcite that provides cross-sentential coreference information for amr graphs. these annotations connect multiple sentence-level graphs via coreference chains, implicit relations and bridging relations. however, an adequate representation of this multi-sentence information in the form of an amr graph has yet to be decided upon. the initial proposal ( @xcite , to merge all nodes in a coreference chain into one node, suffers from significant information loss. without a consistent multi-sentence graph representation, amr's standard smatch metric @xcite cannot be used for evaluation and comparison. the smatch scores differ greatly depending on how the coreference information is added to the amr graphs. additionally, the smatch algorithm is exceedingly slow in its original form when run over multi-sentence graphs. these limitations in multi-sentence amr graph representation and evaluation get in the way of research efforts in this area. without a consistent representation and evaluation mechanism, it is impossible to draw meaningful comparisons between different approaches on the task. in this work, we present a simple and non-lossy multi-sentence amr graph representation as well as a modification of the smatch algorithm that allows efficient and consistent evaluation. 1 our chief contributions are: ## multi-sentence amr corpus multi-sentence amr corpus (henceforth ms-amr) is part of amr3.0 release and was introduced by @xcite . while a number of works have used document-level amr graph creation for downstream tasks such as summarization @xcite , it is the only dataset currently with manual annotations of amr document graphs. it contains 284 documents in its train split and 9 documents on test split, annotated over 8027 gold amrs in total. out of the 284 train documents, 43 are annotated twice; in this work we refer to these double annotation documents as our development set. coreference chains were annotated as clusters of coreferent nodes in gold amr graphs, rather than as coreferent spans of text: this is what allows evaluation with smatch. such an annotation also precludes evaluation of system predictions using more traditional coreference evaluations, as there is no notion of mention spans to use when comparing mentions. the ms-amr corpus also annotated implicit relations, as shown in the blue arg4 edge in figure 1 , whenever a numbered argument of a predicate was semantically identifiable in another sentence. similarly, the ms-amr corpus annotates bridging mentions, such as part-whole or set-member relations. both implicit relations and bridging relations can be seamlessly added as cross-sentential edges. coreference chains form the majority of crosssentential relations annotated in ms-amr. unlike implicit and bridging relations, there is no straight forward way of incorporating coreference chains into a multi-sentence amr graph. next section discusses the challenges involved and proposes a solution. ## document-level amr representation as discussed in the past section, majority of ms-amr annotations take the form of coreference chains over sentence-level gold amr graphs. in this work, we propose a representation docamr that incorporates these chains into multi-sentence amr graphs. the motivation for having such a representation is two-fold. first, representing a multi-sentence document as a single amr graph will allow downstream applications to treat the whole document as single entity, much like the amr graphs for sentences. second, it is crucial to have a consistent representation for the purpose of evaluating ms-amr predictions. smatch -the standard evaluation metric for amr -operates over graphs. we can use this metric for ms-amr evaluation only if the system outputs and the gold graphs adhere to the same document-level graph representation. intrasentential coreferences in sentence-level amr are annotated with what one might call manual full merge -i.e., given multiple mentions in the same sentence (such as a named entity and a pronoun), a human annotator would pick a single informative mention, and merge all other information into the root of the subgraph representing that mention. language being efficient, it is very rare for multiple non-reduced mentions to occur in the same sentence, therefore it was a trivial task for human annotators. ms-amr human annotations do not perform manual merge -they only indicate the nodes in multiple existing sentence-level graphs that participate in a coreference cluster. ## experiments and results we use docamr along with our efficient implementation of smatch to assess the quality of two document-level amr parsing systems. first, we develop a pipeline system combining a topperforming amr parser @xcite . all results are on our best performing pipeline system. r is the number of random restarts. 'original' smatch runs out of memory for @xmath0. @xcite . we also provide the pipeline results with corenlp's neural coreference resolution system @xcite v4.3.2 for additional point of comparison. second, we reevaluate the past best system of @xcite -this is also a pipeline approach combining amr parser of @xcite with allennlp coreference resolution system. foot_7 ## conclusion we have presented docamr, a graph representation for document-level amr graphs based on coreference annotations. relative to the original sentence-level graphs, docamr removes redundancy without information loss. we modified the implementation of smatch to take advantage of sentence provenance to efficiently search for node alignments when comparing two document-level graphs. finally, we reported results for a documentlevel parsing pipeline that can serve as a strong baseline for future work on this task.