When Does Meaning Backfire? Investigating the Role of AMR s in NLI (2025)

Abstract: Natural Language Inference (NLI) relies heavily on adequately parsing the semantic content of the premise and hypothesis.In this work, we investigate whether adding semantic information in the form of an Abstract Meaning Representation (AMR) helps pretrained language models better generalize in NLI. Our experiments integrating AMR into NLI in both fine-tuning and prompting settings show that the presence of AMR in fine-tuning hinders model generalization while prompting with AMR leads to slight gains in GPT-4o.However, an ablation study reveals that the improvement comes from amplifying surface-level differences rather than aiding semantic reasoning. This amplification can mislead models to predict non-entailment even when the core meaning is preserved.

## introduction since the advent of large language models (llms), there has been ongoing debate about the utility of symbolic representations such as abstract meaning representations (amrs; @xcite in (llm-based) pipelines and existing nlp tasks. while some studies report limited or negative impact of amrs on mainstream nlp tasks @xcite , recent work has demonstrated their value in specific applications, such as syntactic simplification @xcite and semantically-controllable text transformation @xcite . perhaps unsurprisingly, incorporating amr has been particularly wellexplored and effective in tasks related to semantics @xcite . natural language inference (nli; @xcite is a popular task in nlp where the solver is given a premise and a hypothesis, and asked to determine whether the hypothesis is true if the 1 we publicly release our code at https://github.com/ aatlantise/advarsarial-nli-amr . premise is true. the label space consists of three labels: entailment if the hypothesis is true, contradiction if the hypothesis is false, and neutral if the truth value of the hypothesis cannot be determined; this can also be condensed in two labels: entailment and non-entailment. as shown in figure 1 "athletes introduced the secretaries" should be entailed by "serious athletes introduced the secretaries." therefore, the label should be entailment because the truth of the premise indicates truth of (or entails) the hypothesis. as a meaning-focused task, nli aligns well with the motivation behind amrs, i.e., to abstract sentence meaning beyond surface form, given nli models' tendencies to adopt shallow heuristics rather than understanding the relationship between the premise and the hypothesis, leading to poor generalization to novel data @xcite @xcite . in this paper, we inves-tigate whether incorporating amrs as additional input -either during (a) fine-tuning or (b) prompting -can encourage models to attend more to abstract meaning, thereby improving generalization and overall performance. as illustrated in figure 1 , we add amrs to either the training data or prompts then evaluate how the addition of amr affects generalization performance. we find that amrs generally hinder performance in both finetuning and prompting settings, with the exception of prompting on hans. however, this improvement appears to stem from amrs amplifying surface-level differences rather than capturing deeper semantic meaning. several prior approaches have incorporated logical representations into nli, for example by combining neural encoders with logical reasoning modules @xcite , training natural logic theorem provers @xcite , extracting phrase correspondences via natural deduction proofs @xcite , or constraining large language models with natural logic inference patterns @xcite . while these works rely on task-specific inference rules or specialized proof systems, our use of amrs differs in that amrs provide a broad, task-agnostic semantic abstraction without requiring dedicated engineering. once an amr parser is available, amrs can be used as direct inputs to pretrained models such as bert (section 3.2 and chatgpt (section 3.3), enabling structured input with minimal task-specific engineering. llms and in-context learning have been used to tackle nli and generalization in it, with mixed results; @xcite show that the content of prompts do not significantly influence llms' performance in nli tasks, while @xcite ; @xcite use chainof-thought reasoning and natural language explanations to improve nli performance and generalization. however, @xcite report that its nli performance is still only comparable to much smaller encoder-only models like bert and roberta @xcite , leaving adversarial nli an ongoing area of research. recent work on amrs has set out to utilize amr graphs for a variety of downstream tasks, including summarization and information extraction (see @xcite ; @xcite for comprehensive overviews). amrs excel in capturing structure-dependent meaning @xcite and have shown particular promise in meaning-sensitive tasks such as debiasing translationese @xcite , style transfer @xcite , and sentence-level manipulation @xcite , especially when used in conjunction with fine-tuned models. to the best of our knowledge, @xcite represents the only prior effort to incorporate amrs into nli, and do so for the purpose of interpretable nli evaluation. they find that metrics based on amr are robust unsupervised representations of premise-hypothesis relationships when used alongside neural representations like bert. ## conclusion in this work, we investigate whether amrs can help plms on the task of natural language inference. specifically, across both fine-tuning and prompting settings, we evaluate whether incorporating amrs improves entailment classification. we find that our implementations of amr integration does not improve performance in finetuning, and only lead to slight gains in zeroshot prompting with gpt-4o. importantly, ablation analyses reveal that these gains are not due to deeper semantic understanding, but rather to amrs exaggerating surface-level differences, which in some cases mislead the model to predict non-entailment where entailment holds. overall, our results suggest that while amrs offer a promising abstraction mechanism, their integration with llms requires careful design to avoid reinforcing shallow heuristics rather than promoting robust reasoning.