X - AMR Annotation Tool (2024)

Abstract: This paper presents a novel Cross-document Abstract Meaning Representation (X-AMR) annotation tool designed for annotating key corpus-level event semantics. Leveraging machine assistance through the Prodigy Annotation Tool, we enhance the user experience, ensuring ease and efficiency in the annotation process. Through empirical analyses, we demonstrate the effectiveness of our tool in augmenting an existing event corpus, highlighting its advantages when integrated with GPT-4. Code and annotations: href{https://anonymous.4open.science/r/xamr-9ED0}{anonymous.4open.science/r/xamr-9ED0} footnote Demo: {href{https://youtu.be/TuirftxciNE}{https://youtu.be/TuirftxciNE}} footnote Live Link: {href{https://tinyurl.com/mrxmafwh}{https://tinyurl.com/mrxmafwh}}

## introduction semantic representations of events play a pivotal role in natural language processing (nlp) tasks, facilitating the understanding and extraction of meaningful information from text. among the various approaches to represent events, semantic role labeling (srl; @xcite ) and abstract meaning representation (amr; @xcite ) have gained significant attention. in this paper, we delve into the realm of semantic event representations, with a particular focus on a method for expanding amr. amr, a graph-based semantic representation, aims to capture the underlying meaning of sentences by breaking them down into atomic concepts and their semantic relationships. each concept in amr is associated with a unique identifier, and the relationships between concepts are represented as labeled edges in a graph. amr has proven to be versatile, serving as a valuable resource for a wide range of nlp tasks such as machine translation, question answering @xcite , and summarization @xcite . its ability to provide a structured, language-independent representation of textual content makes it an essential tool in the nlp toolkit. however, despite its many merits, current amr techniques are not without limitations. one of the primary challenges lies in linking temporal relations and entity coreference across sentences and documents. this limitation hinders the comprehensive understanding of text, as it often fails to capture the intricate interplay between events and entities that span multiple contexts. this issue becomes particularly pronounced in scenarios involving cross-document event coreference, where events mentioned in one document need to be linked to events in other documents for a coherent understanding of a larger narrative. to illustrate the challenge of coreference across documents, consider the following example: two news articles discuss a corporate acquisition. in one article, the event is described as "company a's purchase of company b on july 1st, 2008" while in another article, it is referred to as "in 7/08 company b was acquired by company a." establishing the coreference relationship between these two descriptions is non-trivial, yet crucial for creating a comprehensive representation of the acquisition event. to specifically address the intricate challenges of cross-document event coreference resolution, our research introduces two significant contributions. firstly, we propose a novel framework x-amr. this framework is an enhancement of the existing amr, specifically designed to overcome the challenges inherent in linking events and entities across different documents. x-amr effectively combines the strengths of amr with the ability to create a more comprehensive and coherent depiction of narratives that span multiple sources. secondly, the development of a specialized interface is another key contribution of our work. utilizing the model-in-the-loop annotation methodology, we have leveraged the customized prodigy annotation tool to augment an existing event coreference dataset, the event coref bank plus (ecb+; @xcite ). this development has facilitaed the annotation of x-amr representations, focusing on the annotation interface and the enhanced x-amr dataset. additionaly, we present an evaluation showcasing the accuracy and efficiency of our approach. our research endeavors to demonstrate the effectiveness of x-amr in addressing the limitations of current sentence level amr, especially in linking temporal relations and entity coreference across sentences and documents. formally, amr are structured as labeled, rooted, directed acyclic graphs, which capture abstract concepts, predicate-argument relationships, and entities found in sentences or utterances. they integrate the semantic content addressed by different representation schemes such as srl, named entities recognition (ner; @xcite ), and coreference resolution into a unified representation. for example, for sentence "hp acquired eypmcs.", the corresponding amr is: the above amr graph captures concepts such as events such as "acquire", named entities such as the hp company, and properties of the entity such as their names as graph nodes and subgraphs. their interrelations between concepts and events are then depicted through labeled edges. events are denoted using propbank rolesets, and semantics rela-tions of the entities and events are specified through numbered arguments and non-core relations from amr's role inventory. for example, in the above acquisition event, the arg0 typically specifies the stereotypical agent of an event and arg1 typically specifies the stereotypical patient of an event. additionaly, amr graphs formalize local temporal information, as shown in the provided example. in the preceding disucssion, we highlighted the expressiveness of amr. however, the expressiveness of amr introduces complexities in amr annotation, historically a significant bottleneck for nlp community. the challenge has been to provide a substantial volume of amr annotations to the data hunger statistical machine learning models given the limitations of available tools. the isi editor, serving as the first amr editor, has supported the amr community for over a decade. despite the efficacy of the isi editor, its learning curve is notably steep for annotators. to make amr annotation more accessible, @xcite developed a new annotation approach. they introduced an amr editor based on coding, complemented by a neural network parser model, to streamline the annotation process. the remarkable progress in large language model-based coding assistance, pioneered by ope-nai and microsoft, is transforming the landscape of program synthesis in software engineering. these models, trained in both natural language and programming languages, excel at completing programs by intelligently integrating code history and human instructions. in a similar vein, camra leverages these large language models (llms) to enhance amr annotation. we are pioneering the extension of llms' capabilities, broadening their application to include more complex tasks such as cross-sentential and cross-document coreference and event linking. this initiative represents a significant step forward in harnessing the power of llms for even more sophisticated and long-distance dependent language processing tasks. ## annotation methodology the annotation workflow, as depicted in figure 1 , comprises of two phases. in the first phase we annotate the roleset ids of the event triggers. then we specify the arguments of the event incrementally. during these two phases, we maintain an arguments store and a model-in-the-loop that queries the store and suggests annotators with the most likely arguments. this store and the model are updated when new events are annotated. next, we discuss the annotation guidelines, the interface, and the model-in-the-loop in the annotation workflow. ## future work the next steps include leveraging the x-amr structures in creating efficient methods for neurosymbolic event coreference resolution (ecr). for example, the x-amr annotations could help in filtering the most pertinent event pairs that can be used with more resource intensive methods for estimating coreference @xcite . another important direction is in the estimation of the quality and cost savings of our methodology in doing ecr annotations. quality measured by the number of ecr links that can be found with the least amount of pairwise event mention comparisons @xcite . ## conclusion in this paper, we have introduced a novel approach for cross-document, corpus-level semantic event extraction utilizing the x-amr framework. to facilitate this process, we have developed a modelin-the-loop annotation tool tailored for x-amr annotation, seamlessly integrated with prodigy. this tool has been employed to curate x-amr annotations by enriching an existing event coreference dataset, with contributions from two annotators. to evaluate the effectiveness of our approach, we have introduced a comprehensive assessment of the predictions, incorporating both the model's output and the assistance of gpt.