A n C ast++: Document-Level Evaluation of Graph-based Meaning Representations (2025)

Abstract: Uniform Meaning Representation (UMR) is a cross-lingual document-level graph-based representation that is based on Abstract Meaning Representation (AMR) but extends it to include document-level semantic annotations such as coreference, modal and temporal dependencies.With recent advancements in UMR annotation efforts, a reliable evaluation metric is essential for assessing annotation consistency and tracking progress in automatic parsing. In this paper, we present AnCast++, an aggregated metric that unifies the evaluation of four distinct sub-structures of UMR: (1) sentence-level graphs that represent word senses, named entities, semantic relations between events and their participants, aspectual attributes of events as well as person and number attributes of entities, (2) modal dependencies that represent the level of certainty that a source holds with respect to an event, (3) temporal dependencies between events and their reference times, and (4) coreference relations between entities and between events. In particular, we describe a unified method TC^2 for evaluating temporal and coreference relations that captures their shared transitive properties, and present experimental results on English and Chinese UMR parsing based on UMR v1.0 corpus to demonstrate the reliability of our metric. The tool will be made publicly available on Github.

## introduction uniform meaning representation @xcite is the latest iteration in a series of semantic representation frameworks beginning with prop-bank @xcite , whose focus on the predicate-argument structure has been inherited by abstract meaning representation (amr) @xcite . the current unified schema of umr features a sentence-level representation similar to amr, albeit with several modifications to promote cross-lingual applicability @xcite . in addition, umr introduces a document-level structure that extends beyond sentence boundaries to capture inter-sentence semantic connections such as coreference as well as modal and temporal dependencies. these include entity and event coreference, temporal relations among events and between events and time expressions, as well as modal dependencies between events and their sources, referred to as conceivers @xcite @xcite . figure 1 shows a sample umr annotation for a short document of 2 sentences as an illustration: in the gold umr graph on the left of figure 1 , the 3 light blue nodes at the top represent ubiquitous abstract concepts, which includes the root node that ensures the graph is single-rooted. auth (author of the text) and dct (document creation time) serve as sub-roots of modal and temporal dependencies respectively. a modal dependency graph (mdg) captures the epistemic certainty and polarity with which the conceivers view events or cite other conceivers @xcite . in the example, the author is certain that the allegation by the activists has already been made (':full-affirmative' edge from auth to 's1a:allege-01'). then activists are partially certain that prisoners are mistreated (':partial-affirmative' edge from 's1a2:activist' to 's1m:mistreat-01'). mdg can be traced by following the red edges in the figure. a temporal dependency graph (tdg) annotates the temporal relations between events and time expressions such as dct @xcite . since all of the events in this example apparently took place in the past, the blue edges originating from dct are labeled ':before' which indicates that they happened before this document was created by the author. coreference chain can be found by identifying nodes linked by edge labels such as ':sameevent' and ':same-entity.' in the sample document, 'alleged' ('s1a:allege-01') and 'allegation' ('s2a:allege-01') from sentence 1 and 2 both refer to the same event. furthermore, the pronoun 'their' ('s2p:person') in the second sentence refers to 'the activists' ('s1a2:activist') in the first sentence. finally, the two sentence sub-graphs rooted at 's1a:allege-01' and 's2g:go-08' respectively show some deviations from their corresponding umr annotations. the purple ':aspect' edges in the figure represent aspect which annotates the internal state of an event regarding whether it is an on-going, finished or habitual event, or simply a state with no changes over the course of action, or possibly something else @xcite . in the example, the mistreatment is described as ongoing (thus 'activity'), whereas the allegation and going un-noticed events have both been completed (hence the 'performance' value). it is also worth pointing out that the explicit support for plurality of nominals as well as person for pronouns visualized with cyan-colored edges. in the era of large language models (llms), the role of symbolic meaning representations such as umr and its precursor, amr, has changed dramatically. prior to llms, datasets annotated with symbolic meaning representations were routinely used to train supervised models as components in pipeline nlp systems such as information extraction (e.g., @xcite @xcite , summarization (e.g., @xcite , machine translation (e.g., @xcite , question answering (e.g., @xcite @xcite , and dialog systems (e.g., @xcite . increasingly, symbolic representations are now used to complement llms by providing structure, consistency, and interpretability @xcite . while llms excel at generating fluent and contextually appropriate responses, they often lack the explicit, systematic framework needed for tasks that require precise semantic representation, cross-linguistic consistency, or detailed reasoning. umr fills these gaps by offering a clear, interpretable framework to represent meaning, which can enhance applications such as multilingual nlp, fact verification, and explainable ai. thus, umr and other symbolic systems work alongside llms to ensure that language understanding is not only fluent and adaptable, but also accurate, transparent, and reliable in scenarios that demand high levels of interpretability and rigor. thanks to the recent release of umr v1.0 @xcite , it is now possible to explore automatic umr parsing @xcite . how-ever, while there are evaluation methods for each of the sub-structures of umr, a unified framework for assessing the quality of the overall umr structure is still lacking. this paper introduces ancast++, a novel tool that implements a metric to measure the accuracy of umr graphs. ancast++ starts by mapping vertices between two graphs using the node alignment approach described in @xcite . based on this alignment, the scores of sentence-level graph as well as the modal, temporal dependencies, and coreference relations are computed before finally producing a micro-average score that aggregates the scores of the four components making up the umr graph. since temporal dependencies and coreference relations both give rise to graph structures that can be further completed via transitive closure, we develop a unified algorithm, t c 2 , to leverage this structural commonality and compute scores for both components. the remainder of this article is organized as follows. in section 2, we briefly describe the node alignment method used in ancast. this is followed by a discussion of the evaluation metric for modal dependencies. we then present t c 2 , a method for computing transitive closures of temporal and coreference relations of umr and calculating the similarity between temporal and coreference subgraphs using the lea-inspired method @xcite . after that, we describe the method for allocating weights to the four components. section 3 provides experimental results on the use of this metric to measure the output of a umr parser. section 4 discusses related work, and section 5 concludes the paper. ## methodology developing an evaluation metric for comparing umr graphs presents two core challenges: (i) comparing transitive and expandable relation sets, specifically temporal and coreference relations within the umr graph, and (ii) assigning appropriate weights to different components of the output to generate a comprehensive score for the graph of the entire document. the first step in comparing two umr graphs with nodes that may potentially have different labels is to establish a node mapping using an alignment algorithm. to tackle this problem, an-cast++ adopts the anchor and broadcast algorithm proposed in ancast @xcite to identify node alignment and produces a score by comparing sentence-level semantic graphsin umr. in this section, we briefly summarize the anchor-andbroadcast algorithm used to obtain node mappings (section 2.1). we then present the sub-metrics for each component of the umr graph: the metric for evaluating sentence-level graphs (section 2.2), the metric for evaluating modal dependencies (section 2.3), and the metric for temporal dependencies and coreference relations (section 2.4). finally, we present how the sub-metrics are aggregated into the full umr metric in ancast++ (section 2.5). ## parsing experiments although umr v1.0 corpus contains annotations in 6 languages 3 , current umr parsing results are limited to english only @xcite . this is due to the modular and pipelined setup of the parser which consists of smaller sub-models trained independently on external annotations that do not exist in low-resource languages. in this work, we adapt this framework to present the first experimental results on chinese umr parsing, despite the absence of temporal dependency dataset for the language. in addition, although @xcite reports a comprehensive macro f1 score of 60.0, this does not account for the disparity in the number of sentences across documents. consequently, strong performance on a short document makes a disproportionately high contribution to the overall f1 score. we therefore report the aggregate macro f1 score weighted by the number of sentences per document, with the comprehensive score now at 51.9 for english. both ms-amr and docamr assume coreferent entities have identical referents and can be merged into clusters. however, umr includes temporal relations, modal dependencies, and subset coreference relations that cannot be clustered similarly. to address this, ancast++ takes a fundamentally different, link-based approach for measuring similarity in umr graphs and handling temporal and coreference relations. multiple evaluation metrics for coreference as a standalone task exist, among which b 3 @xcite and ceaf @xcite consider the overlap between nodes (mentions), while muc @xcite and blanc @xcite focus on the links between them. however, most still suffer from being uninterpretable or exploitable in extreme cases, leading to skewed evaluation results in certain scenarios as explained in @xcite . one of the best mitigation methods has been to get an average of multiple metrics outputs as is done in the commonly adopted conll metric @xcite ). not all coreference metrics extend well to temporal relations. although both coreference and temporal relations are transitive, temporal relations cannot form clusters as coreference relations typically do. indeed, even some coreference relations resist plausible clustering. therefore, we adopt the link-based lea approach (moosavi and strube, 2016), as it readily generalizes to temporal relations. early temporal evaluation methods used transitive closures @xcite , but differing opinions about the relevance of certain relations led later work to emphasize core relations or minimal graphs instead @xcite . @xcite applied temporal closure solely to verify explicit annotations rather than comparing two closed graphs. other approaches simply counted annotated triples @xcite . in contrast, ancast++ evaluates fully closed graphs, providing a uniform evaluation framework for both temporal and coreference relations. ## conclusion we present ancast++, an aggregated evaluation tool that implements intuitive metrics encompassing sentence-level annotation, modal dependencies, temporal and coreference relations in the umr graph. ancast++ also includes a novel t c 2 algorithm that unifies the evaluation of temporal and coreference relations using their transitive closures. this represents a significant improvement over the previous scattered metrics in terms of the multifacet annotations on document level semantic content. tructure for uniform meaning representations". any opinions, findings, conclusions or recommendations expressed in this material do not necessarily reflect the views of nsf.