“You Are An Expert Linguistic Annotator”: Limits of LLM s as Analyzers of A bstract M eaning R epresentation (2023)

Abstract: Large language models (LLMs) demonstrate an amazing proficiency and fluency in the \textit{use} of language. Does that mean that they have also acquired insightful linguistic knowledge \textit{about} the language, to an extent that they can serve as an “expert linguistic annotator’? In this paper, we examine the successes and limitations of the GPT-3, ChatGPT, and GPT-4 models, focusing on the Abstract Meaning Representation (AMR) parsing formalism (Banarescu et al., 2013), which provides rich graphical representations of sentence meaning structure while abstracting away from surface forms. We compare models’ analysis of this semantic structure across two settings: 1) direct production of AMR parses based on zero- and few-shot examples, and 2) indirect partial reconstruction of AMR via metalinguistic natural language queries (e.g., “Identify the primary event of this sentence, and the predicate corresponding to that event.”). Across these settings, we find that models can reliably reproduce the basic format of AMR, as well as some core event, argument, and modifier structure - however, model outputs are prone to frequent and major errors, and holistic analysis of parse acceptability shows that even with few-shot demonstrations, models have virtually 0% success in producing fully accurate parses. Eliciting responses in natural language produces similar patterns of errors. Overall, our findings indicate that these models out-of-the-box can accurately identify some core aspects of semantic structure, but there remain key limitations in their ability to support fully accurate semantic analyses or parses.

## introduction llms in recent years have revolutionized artificial intelligence, showing advanced proficiency and fluency in the use of language, and appearing to possess high levels of expertise and analytical capability across a wide variety of specialized domains. observation of these capabilities has raised important questions about the extent, robustness, and limitations of the knowledge and analysis abilities of these models in specialized domains. ## related work a large body of work has examined various aspects of syntactic and semantic capabilities in language models (c.f. @xcite , indicating that llms show strong knowledge of syntactic structure, while semantic capabilities are more mixed. nonetheless, llms have also been used for fewshot semantic parsing with some success. in particular, @xcite and @xcite find that few-shot learning in gpt-3 and codex produces semantic parses that outperform baselines with comparable training sizes. these semantic parsing datasets, which focus on producing database queries in particular domains, are less complex and domain-general than amr, but the results suggest that llms should contain aspects of the knowledge needed to analyze semantic structure. as for amr, pre-trained transformer models have helped to advance the state of the art in amr parsing, with recent amr parsers building on the foundation of models like bart @xcite @xcite . this indicates that pre-trained models may also pick up on representational capabilities relevant for supporting amr. though these prior works are suggestive that llms and pre-trained transformers capture certain aspects of linguistic structure, it is not clear from existing results how detailed or reliable llms' ability to analyze meaning structure may beformalisms used for prior few-shot semantic parsing are simpler and more domain-specific than amr, and the supervised fine-tuning of bart for amr parsing obscures the details of what original knowledge may have been contained in the pre-trained model. we directly examine pre-trained models' ability to produce amr information, and we do so across a number of potentially productive zero-and fewshot settings for maximum insight about model capabilities. we also prioritize fine-grained, manual analysis of models' accuracies at multiple levels of amr information, in order to provide more detailed insights into model capabilities. ## zero-shot amr parsing given findings of superior zero-shot performance in a wide variety of domains @xcite , we begin by testing models' zero-shot capability for generating amr graphs directly. instructions and examples for amr annotation are widely available online, so it is reasonable to imagine that models may learn to do this task zero-shot as well. we input to the model the target sentence and the simple instruction "provide an amr (abstract meaning representation) parse for this sentence." for chatgpt and gpt-4, we include the system message "you are an expert linguistic annotator." our goal here is to use clear and fair prompts that allow us to assess model capabilities and limitations. we do not do elaborate prompt engineering, but take the stance that if a prompt is sufficiently clear, then a failure to perform the task is simply a failure-reliance on particular prompt phrasing or structure is an indication of model brittleness. since the zero-shot parses fail often even at the basic levels, we limit to level 1 analysis for this setting. results are in the top segment of the outputs indicate clearly that these llms have been exposed to amr parse annotations in their pre-training, and have managed to learn surface characteristics of amr structure: we see that for all models a majority of outputs (>70%) show basic amr format despite the absence of any demonstration in the prompt. comparison between publicly annotated sentences and newly annotated sentences from 2023 ( §b) also shows no noteworthy difference, indicating that output quality is not reliant on presence of test amr annotations in pre-training. however, beyond the basic form, all models show frequent and substantial errors in the parsing. the parses identify the correct top node only 20-40% of the time, reflecting routine failures to incorporate clausal and discourse relations that amr often captures in the top node-and even with the more relaxed criterion of identifying the main event relation, llms succeed in only about half of the parses (40%). when we consider the viability of the full structure as an appropriate meaning representation of the sentence, none of the models produce any fully acceptable amr parse. these results suggest that out-of-the-box, zero-shot llm capabilities are limited primarily to mimicking surface format of amr representations, with understanding of the linguistic functions and phenomena being beyond their zero-shot capabilities. ## parsing with few-shot demonstrations given that zero-shot parsing shows non-trivial limitations across all models, we next test how parses improve with few-shot demonstrations of amr parsing. we use the same instructions (and, in chatgpt and gpt-4, system message), but we now include the specification "i will first show some examples." followed by five example sentences with corresponding amr parses, selected based on simfor few-shot parses, we apply both level 1 ( for level 2 analysis, we see that models have limited reliability in identifying a given event's arguments and modifiers (40-50%) or argument modifiers (10-40%). additionally, just under half of parses include at least one spuriously-identified argument or modifier ("extra mods"). qualitative analysis indicates that models make diverse errors that can occur at any level of amr structure, though they show the most reliable accuracy in representing event-argument triplets corresponding to subject-verb-object structures. see §e for additional examples and discussion on these points. ## metalinguistic nl responses by far the most thoroughly trained format for llms is that of natural language, so we next explore models' ability to use natural language to identify and describe the abstract meaning structure relevant for amr, via prompting for metalinguistic information about the target sentence. to do this, we formulate a natural language prompt instructing the model to identify and break down aspects of the sentence meaning structure corresponding to components of amr, similar to the process that an amr annotator would use. our prompt for this setting is shown in §d. in this setting, the prompt asks for a breakdown of events, arguments, and event/argument modifiers, but does not elicit enough information to enable complete parses. for this reason, we focus on our level 2 criteria for analyzing these outputs. results are shown in the bottom of comparing parse vs nl output side-by-side inspection of parse and nl outputs supports the conclusion that, although errors are not identical, the two output formats may reflect real patterns in models' analytical capabilities. an illustration can be seen in figure 1 , which shows gpt-4 few-shot parse and zero-shot nl outputs for the sentence "he woke to an angry house and darkness in the windows". this is a simple sentence, but the argument structure of the verb woke lacks the simple subject/object structure that models succeed at most often-and perhaps for this reason in both output formats the model misinterprets the sentence to include two separate events (missing the fact that the angry house and darkness in the windows is a single argument of the waking event) and creating nonsensical argument and modifier structures as a result of the mistaken analysis. additional sideby-side examples are included in figures 5 , 6 , and 7 in the appendix, further illustrating similarities between parse and nl outputs, and supporting the possibility that the observed errors in these outputs reflect real underlying analytical limitations rather than artifacts of instructions or output format. ## smatch comparison to anchor our results relative to an existing metric, we obtain smatch scores for our five-shot gpt-4 3 though not included in amr3 2023 lpp amrbart 0.78 0.79 0.75 gpt-4 (no fixes) 0.42 0.26 0.42 + auto fixes 0.48 0.40 0.47 + manual fixes 0.51 0.47 0.51 we are confident that additional fiddling and clever manipulations can further improve the outputs of these models, at least on certain dimensions. however, we present these results as a current status report and reality check to counterbalance frequent claims focused on widespread success and intelligence of these models out-of-the-box. we look forward to continuing work to better understand the fundamental strengths and limitations of these models in this domain, and to improve the reliability of semantic analysis capabilities achievable through collaboration with these models.