AMR 4 NLI : Interpretable and robust NLI measures from semantic graphs (2023)

Abstract: The task of natural language inference (NLI) asks whether a given premise (expressed in NL) entails a given NL hypothesis. NLI benchmarks contain human ratings of entailment, but the meaning relationships driving these ratings are not formalized. Can the underlying sentence pair relationships be made more explicit in an interpretable yet robust fashion? We compare semantic structures to represent premise and hypothesis, including sets of *contextualized embeddings* and *semantic graphs* (Abstract Meaning Representations), and measure whether the hypothesis is a semantic substructure of the premise, utilizing interpretable metrics. Our evaluation on three English benchmarks finds value in both contextualized embeddings and semantic graphs; moreover, they provide complementary signals, and can be leveraged together in a hybrid model.

## introduction natural language inference (nli) and textual entailment (te) assess whether a hypothesis (h) is entailed by a premise (p). systems have various interesting applications, e.g., the validation of automatically generated text @xcite . recent systems make use of neural networks to encode h and p into a vector and thereupon make a prediction @xcite . while this can provide strong results when such systems are trained on largescale training data, the overall decision process is not transparent and may rely more on spurious cues than on informed decisions @xcite . we aim to develop more transparent alternatives for nli prediction, and therefore compare representations and metrics to predict entailment. figure 1 gives an intuition of how 5 different sentences overlap in meaning. representing each sentence with a semantic structure, we assume that, by and large, the semantic elements of an entailed sentence should be contained within the premise. these considerations trigger three interesting research questions that we will investigate in this paper: rq1. how to characterize a semantic structure? rq2. how to determine/measure what is a substructure? rq3. is there a suitable and interpretable structure and measure that help to make nli judgments more robust, or more accurate? to assess rq1, we test three options: token sets, sets of contextualized embeddings, or graph-based meaning representations (mrs). as a meaning representation, we select abstract meaning representation (amr; @xcite , using automatic amr parses of the nli sentences. to assess rq2, we test different types of metrics that are designed or adapted to measure entailment on the selected structures, inspired from research on, e.g., mt evaluation and mr similarity. one of our key goals is to investigate whether it is possible to accurately capture relevant semantic substructure relationships via meaning representations. finally, we show that we can positively answer all aspects of rq3: first, besides their enhanced interpretability, unsupervised semantic graph metrics are more robust and generalize better than fine-tuned bert. second, importantly, we show that they are high-precision nli predictors, a property that we exploit to achieve strong nli results with a simple decomposable hybrid model built from a fine-tuned bert on the one hand, and a semantic graph score on the other. code and data are available at https://github.com/flipz357/amr4nli . ## related work textual entailment automatic approaches for this task date back to, at least, @xcite , who introduced a shared task for entailment classification. since then, we can distinguish many different kinds of systems for addressing the task (androutsopoulos and malakasiotis, 2010), for instance, based on logics @xcite or string-and tree-similarity @xcite , or graph matches of semantic frames and syntax @xcite that aim in a similar direction as us. recent releases of largescale training corpora, such as snli @xcite , or mnli @xcite can be exploited for supervised training of strong classifiers, e.g., by fine-tuning a bert language model @xcite . however, trained systems tend to suffer from the 'clever hans' effect and fall prey to spurious cues @xcite , such as position @xcite or even gender @xcite . this can lead to undesired and peculiar nli system behavior. @xcite show that supervised nli systems can make many correct predictions solely based on p, without even seeing h. in our work, we want to test more transparent ways of rating entailment. ## conclusion we find that metrics defined on advanced semantic representations are useful predictors of entailment. this is especially true for metrics performing asymmetric measurements on graph-structured meaning representations and sets of contextualized embeddings. interestingly, meaning representation-based metrics offer advantages over strong embeddingbased metrics beyond just interpretability: while showing similar performance as bertscore, they are more robust than fine-tuned bert and offer high-precision predictions. with this, we show that linguistic and neural representations can complement each other in a hybrid model, leading to substantial improvement over both untrained and trained neural approaches.