AMR Quality Rating with a Lightweight CNN (2020)

Abstract: Structured semantic sentence representations such as Abstract Meaning Representations (AMRs) are potentially useful in various NLP tasks. However, the quality of automatic parses can vary greatly and jeopardizes their usefulness. This can be mitigated by models that can accurately rate AMR quality in the absence of costly gold data, allowing us to inform downstream systems about an incorporated parse’s trustworthiness or select among different candidate parses. In this work, we propose to transfer the AMR graph to the domain of images. This allows us to create a simple convolutional neural network (CNN) that imitates a human judge tasked with rating graph quality. Our experiments show that the method can rate quality more accurately than strong baselines, in several quality dimensions. Moreover, the method proves to be efficient and reduces the incurred energy consumption.

## introduction the goal of sentence meaning representations is to capture the meaning of sentences in a well-defined format. one of the most prominent frameworks for achieving this is abstract meaning representation (amr) @xcite . in amr, sentences are represented as directed acyclic and rooted graphs. an example is displayed in figure 1 , where we see three equivalent displays of an amr that represents the meaning of the sentence "the baby is sleeping well". in amr, nodes are variables or concepts, while (labeled) edges express their relations. among other phenomena, this allows amr to capture coreference (via re-entrant structures) and semantic roles (via :arg n relation). furthermore, amr links sentences to kbs: e.g., predicates are mapped to propbank @xcite , while named entities are linked to wikipedia. from a logical perspective, amr is closely related to first-order logic (fol, see @xcite for translation mechanisms). currently, amrs are leveraged to enhance a variety of natural language understanding tasks. e.g., they have enhanced commonsense reasoning and question answering @xcite , machine translation @xcite ), text summarization @xcite and paraphrasing @xcite . however, there is a critical issue with automatically generated amrs (parses): they are often deficient. these deficiencies can be quite severe, even when high-performance parsers are used. for example, in figure 2 , a neural parser @xcite conducts several errors when parsing without a functioning economy the whole country may destabilize. e.g., it misses a negative polarity and classifies a patient argument as the agent by failing graph representation computer processing human understanding well-defined triples (e.g., gnn) graph visualization (short sentences) penman, linearized string (e.g., lstm) penman, indents (this work) to see that destabilize here functions as an ergative verb (parser: the country is the causer of destabilize; correct: the country is the object that is destabilized). in sum, the parse has misrepresented the sentence's meaning. 1 however, assessing such deficiencies via comparison against a gold reference (as in classical parser evaluation) is often infeasible in practice: it takes a trained annotator and appr. 10 minutes to manually create one amr @xcite . to mitigate these issues, we would like to automatically rate the quality of amrs without the costly gold graphs. this would allow us to signal downstream task systems the incorporated graphs' trustworthiness or select among different candidate graphs from different parsing systems. to achieve this, we propose a method that imitates a human rater, who is inspecting the graphs. we show that the method can efficiently rate the quality of the amrs in the absence of gold graphs. the remainder of the paper is structured as follows: in section 2, we outline our idea to exploit the textual multi-line string representation of amrs, allowing for efficient and simple amr processing while preserving vital graph structure. in section 2.2, we instantiate this idea in a lightweight cnn that predicts the quality of amr graphs along multiple dimensions of interest. in our experiments (section 3), we show that this framework is efficient and performs better than strong baselines. our code is available at https://github. com/flipz357/amr-quality-rater . ## amr as image with latent channels in this section, we first motivate to treat amrs as images with latent channels in order to rate them efficiently. second, we briefly describe the task at hand: rating the quality of amr graphs in the absence of gold graphs. finally, to solve this task, we create a lightweight cnn that evaluates amr quality in multiple dimensions of interest. the penman notation and its (hidden) advantages the native amr notation is called pen-man-notation or sentence plan language @xcite . provably, an advantage of this notation is that it allows for secure amr storage in text files. however, we argue that it has more advantages. for example, due to its clear structure, it allows humans a fairly quick understanding even of medium-sized to large amr structures (figure 3 , left). on the other hand, we argue that a graphical visualization of such medium-sized to large amrs (figure 3 , right) could hamper intuitive understanding, since the abundant visual signals (circles, arrows, etc.) may more easily overwhelm humans. moreover, in every display, one would depend on an algorithm that needs to determine a suitable (and spacious) arrangement of the nodes, edges and edge labels. it may be for these reasons, that in the amr annotation tool foot_0 , the graph that is under construction is always shown in penman notation to the human user. in sum, we find that the indented multi-line pen-man form possesses three key advantages ( amr as image to preserve graph structure figure 4 describes our proposed sentence representation treatment. after non-degenerate amr graph simplification (more details in preprocessing, 3.1) , we first project the penman representation onto a small grid ('image'). each amr token (e.g., a node or an edge) is represented as a 'categorical pixel'. second, φ adds latent 'channels' to the categorical pixels, which can be learned incrementally in an application. in other words, every amr token is represented by a fixed-sized vector of real numbers. these vectors are arranged such that the original graph structure is fully preserved. ## experiments in this section, we first describe the data, changes to the data that target the reduction of biases, and the baseline. after discussing our main results, we conduct further analyses. (i), we study the effects of our data-debiasing steps. (ii), we assess the performance of our model in a classification task (distinguishing good from bad parses). (iii), we assess the model performance when we only provide the candidate amr and the sentence (dependency tree ablation). (iv), we provide detailed measurements of the method's computational cost. ## related work quality measurement of structured predictions since evaluating structured representations against human annotations is costly, systems have been developed that attempt an automatic quality assessment of these structures. due to its popularity, much work has been conducted in machine translation (mt) under the umbrella of quality estimation (qe). qe can take place either on a wordlevel @xcite , sentence-level @xcite , or document-level @xcite . the conference on machine translation (wmt) has a long-standing workshop and shared task-series on mt quality assessment @xcite @xcite @xcite @xcite . quality estimation for neural language generation has been investigated, i.a., by @xcite , and recently by @xcite , who design a model that jointly learns to rate and rank generations, or by @xcite , who predicts pair-wise preferences for generated summaries. furthermore, automatic techniques for the quality assessment of syntactic parses have been proposed. for instance, @xcite formulate the task as a single-variable regression problem to assess the quality of constituency trees. a major difference to our work is that they try to assess the performance of a single parser, while we aim at a parser-agnostic setting where candidate parses stem from different parsers. similarly, @xcite predict a binary label that reflects whether the tree-quality lies above a certain threshold (or not). when multiple candidate parses are available, tree ranking methods @xcite may also be interpreted as some form of parse quality assessment (see @xcite for a recent overview). compared with assessing the quality of (abstract) meaning representations, judging about syntactic trees perhaps is a conceptually slightly simpler task, since the syntactic graphs are more directly grounded in the sentence foot_5 , and therefore it may be easier to judge whether graph components are correct, redundant, missing, or false. in comparison to mt, automatic quality assessment of meaning representations is insufficiently researched. @xcite propose an lstm based model that performs a multi-variate quality analysis of amrs (constituting the baseline which we compared against). we believe that quality estimation approaches may also prove valuable for other meaning representation formalisms (mrs), such as, e.g., discourse representations @xcite @xcite or universal semantic dependencies @xcite . for example, since the manual creation of mrs is a notoriously laborious task, automatic quality assessment tools could assist humans in the annotation process (e.g., by serving as a cheap annotation quality check or by filtering automatic parses in active learning). amr metrics when a gold graph is available, it can be used to compute the canonical amr metric smatch @xcite that assesses matching triples. furthermore, @xcite have extended smatch to inspect various aspects of amr. in this work, we have shown that our model can predict the expected outcomes of these metrics in the absence of the gold graph. recently, more amr metrics have been proposed, for example the bleu-based @xcite sembleu metric @xcite , sema (anchiêta et al., 2019) or s 2 match @xcite , a variant of smatch. we plan to extend our model such that it also predicts these metrics. amr parsing recent advances in amr parsing have been achieved by parsers that either predict latent alignments jointly with nodes @xcite , or by transducing a graph from a sequence with a minimum spanning tree (mst) decoding algorithm @xcite , or by focusing on core semantics in a top-down fashion @xcite , or by performing auto-regressive decoding with a graph encoder @xcite . other approaches apply statistical machine translation @xcite or sequence-to-sequence models, which tend to suffer from data scarcity issues and need considerable amounts of silver data to improve results @xcite . previously, alignment-based pipeline models have proved effective @xcite or transition-based approaches that convert dependency trees step-by-step to amr graphs @xcite @xcite . ## conclusion in this work, we have developed an approach to rate the quality of amr graphs in the absence of costly gold data. our model imitates a human judge that is confronted, 'on paper', with the amr in its native multi-line penman format. we saw how this setup allowed efficient amr processing with convolutions. our experiments indicate that the method rates amr quality more accurately and more efficiently than previous work.