CAMRA : Copilot for AMR Annotation (2023)

Abstract: In this paper, we introduce CAMRA (Copilot for AMR Annotatations), a cutting-edge web-based tool designed for constructing Abstract Meaning Representation (AMR) from natural language text. CAMRA offers a novel approach to deep lexical semantics annotation such as AMR, treating AMR annotation akin to coding in programming languages. Leveraging the familiarity of programming paradigms, CAMRA encompasses all essential features of existing AMR editors, including example lookup, while going a step further by integrating Propbank roleset lookup as an autocomplete feature within the tool. Notably, CAMRA incorporates AMR parser models as coding co-pilots, greatly enhancing the efficiency and accuracy of AMR annotators.

## introduction abstract meaning representation (amr) stands as one of the most widely embraced formalisms for deep lexical semantic representation within the nlp community. it effectively captures the lexical semantics present in multiple sentences by employing a directed, acyclic graph, wherein graph nodes form predicate-argument structures locally in neo-davidsonian fashion @xcite . amr can address both superficial semantic inquiries, encompassing aspects like "who did what to whom, when, where, and how," as well as the intricate relationships between various events and states. beyond these merits, amr offers an invaluable advantage through its transparent symbolic representation of the semantics inherent in natural language text, significantly benefiting tasks reliant on semantic inference and necessitating interpretability. over the past decade, nlp researchers have meticulously transcribed tens of thousands of natural sentences into amr graphs @xcite @xcite @xcite , providing a critical source for the statistical machine learning approach to semantic parsing. while these resources are invaluable, producing amrs is difficult, involving many sub-tasks, such as the annotation of nouns/named-entities, predicate-argument dependencies, co-reference resolution, discourse connectives, negation, and temporal relations. on top of this, annotators would need to be thoroughly trained to navigate a complex annotation tool interface in the process. traditional amr editors typically begin by having the annotator construct a root node and then add additional nodes as children through graph traversal. nodes are added either through a dashboard made up of a combination of buttons, menus, and entry fields or, through a command-line-like interface with a series of learned commands. both versions add yet another layer of learning complexity to the already intricate amr structure. however, this is not simply an amr problem but a problem for all semantic annotation tasks that involve complex structural layers of annotation. we present an example amr in figure 1 for the sentence "the boy must not go." in an amr graph, predicates and their corresponding arguments are represented by nodes. in this example, the go-02 predicate 2 @xcite has one argument, which is boy. amr specifies the role of each argument with labeled edges. core roles, like stereotypical agent, patient, and thematic role, are typically denoted by arg0, arg1, and arg2, respectively. other non-core roles, which are usually predicate-specific, are directly labeled with their names, such as location, direction, time, and duration. amrs can be expressed in various formats, but graphically annotating their complex structure is impractical. to address this researchers adapted penman notation (goodman, 2019, 2020), which represents graph structures using bracketing syntax. labeled edges are encoded with the preceding colon symbol, and opening brackets indicate new amr nodes. terminal nodes are denoted by closing parentheses. advancements in large language model-based coding assistance, like codex @xcite and copilot by openai and microsoft have been revolutionizing program synthesis for software engineering tasks. these models are trained for both natural languages and programming languages, enabling them to intelligently complete programs based on code history and human instruction. drawing inspiration from code-completion approaches, we take a similar path by integrating an amr parser model alongside a human annotator. this unique combination allows us to streamline the amr construction process, handling easier yet tedious tasks like named entity sub-graph construction through the parsing assistant. at the same time, more intricate annotation decisions, such as predicate sense distinction, discourse connections, and co-reference resolutions, can be moderated by the annotators themselves, ensuring a balanced and effective approach to amr annotation. we summarize our contributions to the semantic annotation task as follows: our highly modularized implementation enables easy swapping of language syntax and assistant models, creating a flexible "programming as annotation" paradigm adaptable to various languages and structures. the two most widely used amr annotation tools are isi amr editor (isi-editor) @xcite and anafora @xcite . both are web-based text annotation tools that focus on different levels of amr annotation. isi-editor is primarily designed for lexical-level amr annotation, while anafora is commonly used to construct document-level amrs based on existing sentencelevel amrs. our work with camra is primarily comparable to isi-editor, as we also focus on sentence-level amr construction. however, it is worth noting that editing cross-sentence relations, such as inter-sentential coreference resolution, can also be accomplished through camra with relative ease. we will later showcase how to use isi-editor in comparison to our approach. the isi editor offers comprehensive support for editing an amr graph through various operators, including top to initiate an amr graph and add to create an amr triplet relation. figure 2 illustrates an example of this functionality foot_1 . in the first view, we enter the add operator in the command field and submit it to activate the action template view. here, we proceed to fill in the new role with the specified head variable, role label, and argument concept node. isi editor processes the template form with verification to ensure the action is valid, resulting in an updated amr displayed in the viewport. in total, annotators have access to 8 core operators that allow them to modify the amr graph in pen-man encoding form with shortcuts for advanced users. additionally, the isi editor provides helpful annotation facilitations, such as the ability to search for existing amr data and perform error checks on demand. these functionalities are organized within a dashboard interface, equipped with menu buttons that trigger specific features. using isi editor becomes a process of sequentially filling small forms. there are other annotation tools available for various complex linguistic-driven tasks, such as the uccaapp @xcite for universal conceptual cognitive annotation @xcite , the brat rapid annotation tool @xcite for universal dependency tree construction, treeeditor for rhetorical structure theory @xcite . however, unlike amr, which can exist without explicit alignment of concept nodes to the natural language surface text, these mentioned tasks are tightly anchored to the surface text. consequently, they require clickselection-based interactions with the user to initiate the annotation process. knowtator is another annotation tool that facilitates ontology construction in protege from text, the ui design of knowtator @xcite also relies on small form filling. furthermore, most syntactic tasks involve a limited number of relationships, typically not exceeding a dozen, in contrast to amr, where the number of rolesets directly corresponds to the number of predicates in a given language. in languages like english, the number of predicates can easily surpass 5000. the unique challenge of annotating amr, coupled with the lack of support for other formalisms, has motivated us to create an annotation tool equipped with a formal language coding environment. this tool aims to enhance the efficiency and accuracy of amr annotation and provide a novel solution to handle its distinctive complexities. recently, significant strides have been made in advancing the development of the model-in-theloop annotation style, aimed at fostering machineassisted human annotations. popular tools, such as prodigy foot_2 and inception @xcite , primarily focus on providing annotation suggestions for text classification, named entity recognition, and entity relation extraction. more recent methodologies have expanded these capabilities to encompass entity and event coreference resolution @xcite . however, the field of machine-assisted annotations for amr remains relatively under-explored. our work endeavors to address this gap, contributing to the enhancement and expansion of this vital aspect of the annotation landscape. ## design and features to enhance the effectiveness of annotator-computer interaction, it is essential for the computer to play an active role in the annotation process, rather than serving merely as a passive typewriter. at the same time, it is important to minimize the need for annotators to frequently shift their attention among different views to memorize local predicate-argument structures temporarily while completing amr annotations. to address these concerns, we have formulated the following design principles: ## discussion the primary user experience factor for our annotation tool is the response time of the parser-based language server. to evaluate this response time in relation to sentence length, we conducted tests using 1000 randomly selected sentences from the training set of ldc2020t02. the results are depicted in figure 6 . the average sentence length among these sentences is 21.72 tokens (tokenized by bart tokenizer), with a standard deviation of 14.76. with and without gpu support, the average response times are 1.62 and 4.47 seconds, respectively, with corresponding standard deviations of 0.86 and 3.32 seconds. we present a box chart illustrating the response time distribution in relation to sentence length (token numbers n), considering the bart model's theoretical inference complexity of o(n 2 ) and the prior distribution of sentence lengths. the testing machine has 2.2ghz intel xeon (r) cpu (24 cores), 256gb ram and a nvidia titan xp gpu (12gb). ## conclusion and future works camra introduces a novel semantics annotation paradigm with considerable potential for enhancement. given the similarity of the parser structure, integrating llm into autocomplete and suggestion output is seamless. we are actively working on finetuning llms to make parsing copilot suggestions more interactive. to assess the language server's impact compared to traditional amr annotation tools, we will conduct a human study involving annotators. furthermore, we aim to expand the pool of potential annotators, serving the dual purpose of broadening our annotator base and supporting computational semantics education. collaborating with nlp communities, we plan to extend support to other formalisms and annotation schemes. moreover, we envision the integration of large language models into the language server, providing more natural language assistance from ai. this advancement could lead to yet another valuable application of large language models, enhancing their inter-pretability and error resilience through the fusion of neural and symbolic approaches. such developments offer exciting possibilities for safer and more innovative applications.