Can LLM s Interpret and Leverage Structured Linguistic Representations? A Case Study with AMR s (2025)

Abstract: This paper evaluates the ability of Large Language Models (LLMs) to leverage contextual information in the form of structured linguistic representations. Specifically, we examine the impact of encoding both short and long contexts using Abstract Meaning Representation (AMR) structures across a diverse set of language tasks. We perform our analysis using 8-bit quantized and instruction-tuned versions of Llama 3.1 (8B), Phi-3, and Mistral 7B. Our results indicate that, for tasks involving short contexts, augmenting the prompt with the AMR of the original language context often degrades the performance of the underlying LLM. However, for tasks that involve long contexts, such as dialogue summarization in the SAMSum dataset, this enhancement improves LLM performance, for example, by increasing the zero-shot cosine similarity score of Llama 3.1 from 66% to 76%. This improvement is more evident in the newer and larger LLMs, but does not extend to the older or smaller ones. In addition, we observe that LLMs can effectively reconstruct the original text from a linearized AMR, achieving a cosine similarity of 81% in the best-case scenario.

## introduction in recent years, large language models (llms) have achieved remarkable success in numerous natural language processing (nlp) tasks, such as machine translation, summarization, and both singleand multi-hop question answering. however, a critical challenge remains: assessing their ability to interpret and utilize meaning from structured representations that encode language in concise and abstract forms. structured semantic representations of text, such as abstract meaning representation (amr) structures and discourse representation structures (drs), have consistently proven effective for reasoning with high-level semantics in text and enhancing performance in challenging structure-aware nlp tasks, particularly those involving long contexts. for instance, prior work has demonstrated that encoding amr structures via text-graph attention can improve long-dialogue summarization performance @xcite . in this paper, we systematically examine the ability of several prominent llms to interpret amrs. in addition, we study how amr-augmented prompting (see fig. 23 ) and amr-only prompting (see fig. 22 ) affect model performance on downstream tasks, that require a deep understanding of context, compared to context-only, i.e., languageonly prompting. unlike previous work, which primarily focuses on developing ad-hoc neural architectures to either improve text regeneration from amrs @xcite or enhance performance on downstream tasks @xcite , this study evaluates the ability of llms to directly interpret linearized (flattended) amrs. here, amr-augmented prompting refers to including the linearized amr of the context in the prompt to evaluate the model's ability to improve its understanding of the original context with support from the amr. amr-only prompting refers to providing the llm with only the linearized amr, without the original language context, to assess the model's ability to infer meaning directly from these semantic representations. our key contributions are as follows: this work provides valuable insights into leveraging amrs to enhance llm performance on downstream language tasks, while also identifying key challenges. we hope that our work will serve as a guide for comprehensively analyzing other structured representations, such as knowledge graphs and discourse representation structures, with respect to how they can be interpreted and utilized by llms. ## related work existing studies have explored various methods for generating text from language and knowledge representation structures, usually by modifying or improving underlying neural architectures. for instance, approaches like text-graph attention @xcite and jointgt @xcite alter the transformer architecture using structured cross-attention to better account for graph properties. prior studies have also demonstrated enhanced open-domain dialogue evaluation by fusing graphencoded amrs into llms via gating mechanism @xcite . others, such as text generation from knowledge graphs using graph transformers @xcite , use graphical representations directly, alleviating the need for graph linearization. these studies have aimed to enhance the quality of generated text using amrs as input through structure-aware methods, semantic aggregation, and other heuristics. however, all of these approaches exhibit the key limitation that the proposed work seeks to address. namely, they focus on modifying model architectures or introducing new architectural components, such as joint graph-text representations and heuristic-based conditioning, which can increase complexity and is less amenable to generalization across domains, tasks, and structures. in contrast, we directly evaluate the inherent capability of existing, general-purpose llms to interpret structured representations like amr. moreover, while most prior studies largely sought to improve the quality of generated text @xcite , the proposed work evaluates the opportunities of amrs to represent context for a wide range of downstream language tasks. ## methodology this section describes amrs, how they are extracted from text, and the pipeline for studying their efficacy in language understanding and reasoning tasks. ## experimental settings our analysis pipeline involves converting text into amr structures and then prompting llms to perform tasks using those amrs, either via amraugmented or amr-only prompting. the baselines for these tasks are simply the language-only (i.e., context-only) prompting results. in this section, we will describe the experimental setup for the tasks outlined in section 3.2. ## results all scores presented in the charts and tables are expressed as percentages. for each task, except for amr-to-text, we present the results of the best performing model in the main paper and report the remaining results in the appendix. for all the zero-shot experiments except amrto-text, 5 different transformer seeds were used to compute 90% confidence intervals. similar performance was recorded across all seeds for zero-shot experiments, resulting in narrow confidence intervals. for all the few-shot experiments except amrto-text, 5 distinct sets of examples were curated from the training set to determine 90% confidence intervals. all the scores presented in the results and tables are averaged across different seeds and sets, if applicable. ## conclusion in this study, we investigated the ability of llms to interpret and leverage amrs. our findings demonstrate that amr-augmented prompting, where the amr of the context is included alongside the original context, significantly improves zero-shot longdialogue summarization for llama3.1 and noticeably improves sentence-level natural language inference for phi3. the improvement from amraugmented prompting is more significant in sam-sum summarization with llama3.1, the largest and most recently developed model among those used in this study, compared to other models and tasks. while amrs did not enhance performance in other tasks, there was still some evidence that llms can extract meaningful information from amrs. this was particularly evident in some of the amr-only experiments, where the models achieved reasonable performance using amrs alone. additionally, we conducted rank-32 lora fine-tuning of llama3.1 on the samsum summarization task, which produced results better than zero-shot but not as effective as few-shot prompting for both context-only and amr-augmented approaches. overall, the experiments suggest that amrs can assist llms in understanding long-term dependencies, key ideas, and events in long texts, as demonstrated by the zero-shot and few-shot samsum summarization experiments. however, including linearized amrs in the prompt appears generally ineffective for tasks involving short contexts.