On the Role of Semantic Proto-roles in Semantic Analysis: What do LLM s know about agency? (2025)

Abstract: Large language models (LLMs) are increasingly used in decision-making contexts, yet their ability to reason over event structure—an important component in the situational awareness needed to make complex decisions—is not well understood. By operationalizing proto-role theory, which characterizes agents via properties such as *instigation* and *volition* and patients via properties such as *change of state*, we examine the ability of LLMs to answer questions that require complex, multi-step event reasoning. Specifically, we investigate the extent to which LLMs capture semantic roles such as “agent” and “patient” through zero-shot prompts, and whether incorporating semantic proto-role labeling (SPRL) context improves semantic role labeling (SRL) performance in a zero-shot setting. We find that, while SPRL context sometimes degrades SRL accuracy in high-performing models (e.g., GPT-4o), it also uncovers an internal consistency between SPRL and SRL predictions that mirrors linguistic theory, and provides evidence that LLMs implicitly encode consistent multi-dimensional event role knowledge. Furthermore, our experiments support prior work showing that LLMs underperform human annotators in complex semantic analysis.

## introduction philosophies of ethics as far back as aristotle @xcite @xcite define moral agents as entities which can be held responsible for their acts, and moral patients as entities which experience the effects of those acts, and are thus objects of concern. in linguistics, psychology, and neuroscience, agency and patiency are studied to understand how individuals perceive, represent, and communicate about actions and their consequences. with llms increasingly used for decision-making, further investigation into their capacities for moral reasoning and judgment has been called for @xcite . additionally, while srl requires the entities in a sentence to be labeled with verb-specific semantic roles (agent of hitting/hitter, patient of hitting/hittee), sprl properties are invariant across verbs. neuroscience research has shown evidence that both verb-specific and verb-invariant event roles are encoded in the brain, and could play complementary roles in processing @xcite . sprl and srl, then, can provide a benchmark for an llm's semantic reasoning capabilities as compared to a human's, a proxy for an llm's situational awareness, and scaffolding for fine-grained interpretability of moral judgments. we ground our experiments in the following research questions: (1) does providing llms with semantic proto-role labels as context for semantic role labeling result in accuracy gains compared to no sprl context?, (2) do errors in semantic role labeling correlate with errors in semantic proto-role labeling?, and (3) how does an llm perform as the srl and sprl tasks increase in complexity? our results enable us to make the following contributions: additionally, while our findings did not produce evidence for positive prompt-level interactions between srl and sprl, we found robust evidence that gpt-4o classifies propbank-style semantic roles consistently with dowty's proto-role theory, suggesting that it encodes a hierarchy of event role structure similar to what has been proposed by linguistic theory. there is also faint evidence of such structures emerging in a much smaller lm, the 3b-parameter version of llama 3.2. our study addresses similar questions at a larger sample size and provides evidence that the findings of @xcite and @xcite generalize to larger data. previous work has found that pre-trained language models like bert implicitly learn semantic properties, including sprl @xcite . stengel-eskin and van durme (2022) investigate the effect of providing sprl context in prompts eliciting interpretations of sentences involving subject control clauses. they find that large language models contain spr property knowledge but do not directly apply it to all situations. to date, however, there has been no fine-grained study of the capabilities of the most recent llms on both sprl and srl, and the relation between them. ## data we evaluate srl on two datasets using the standard test splits: the first one (spr1 @xcite , @xmath0, but contains sprl annotations, which allows a fine-grained analysis of the effects of each proto-role property. the second dataset (an srl test split from ontonotes foot_1 , @xmath1. we use ontonotes to ascertain whether our analysis from spr1 might generalize to a larger dataset with a more diverse set of topics and semantic and syntactic constructions. sprl results are typically reported on two english-language datasets: spr1 and spr2 @xcite . spr1 contains 4,912 wall street journal sentences from propbank annotated by a single annotator based on a set of 16 proto-role properties. 9,738 arguments were annotated for the likelihood (on a likert scale from 1 to 5) that a property holds for that argument. spr2 contains 2,758 english web treebank @xcite sentences annotated for a smaller set of 14 properties using a revised, streamlined protocol with two-way redundancy in annotation. the sets of properties in spr1 and spr2 are slightly different from one another, and neither maps one-to-one to dowty's original 10 proto-role entailments. previous work @xcite @xcite , formulates sprl as a 16 (spr1) or 14 (spr2) way multi-label binary classification problem and map likert labels {1, 2, 3} to 0, and {4, 5} to 1. previous work additionally maps judgments labeled "inapplicable" to 0. in our work, we use standard train/dev/test splits provided in the data where applicable: wherever we report results, those are based off of test sentences. ## experimental setup we experiment on a variety of open-weight models ranging from 3 billion to 8 billion parameters: llama 2 and 3 @xcite , qwen2.5 @xcite , and tülu 3 (post-trained using llama 3.1 as a foundation; @xcite . we also experiment on openweight, open-data models olmo 1 and 2 @xcite and pythia @xcite . because the olmo and pythia authors release their data to the public, we could confirm that those models were not trained on data contaminated with the data we test on @xcite . finally, we perform a broad range of experiments that require a longer context window and a more complex array of tasks on @xcite . we evaluate on a variety of prompt templates with and without certain context. our main focus is the model's ability to reason over the semantics of a sentence in a zero-shot setting, and we use propbank and proto-role theory to validate whether its reasoning is consistent. we utilize three different zero-shot prompt templates for srl, and one prompt for sprl alone: further experiment details can be found in §a. ## results and discussion one-arg prompt variant results on spr1 are aggregated in why did the all-args and pipeline (oracle) prompts yield better results than one-arg? one possible explanation is that all of the arguments being in the prompt context for the former two variants, as opposed to only one in the latter, forces the accuracy difference prompt variant all-args one-arg pipeline-oracle model to consider all arguments at once and prevent it from assigning, for example, agent-of-action to two different arguments, but more experimentation would be necessary to isolate this effect. the sprl results ( lms that perform well on sprl tend to perform well on srl, with two notable exceptions: qwen2.5 performed the worst on sprl while having the highest srl accuracy out of all the other small lms. on the other hand, llama 3.2 (3b) exhibited competitive sprl performance while lagging behind in srl. interestingly, models tend to perform better on spr2, opposite the trend in previous work, which has been explained by the spr2 training set containing (1) less data than spr1 or (2) more complex predicates and arguments. that zero-shot prompting methods yield better results on spr2 than spr1 seem to confirm that (1) was the case. in general, all zero-shot prompts we tested underperformed previous work, suggesting that fine-tuning or fewshot prompting would be a more fruitful method for utilizing llms for this task. ## conclusion and future work our study investigated how llms reason about agency and patiency by examining the interplay between semantic proto-role labeling (sprl) and semantic role labeling (srl). we found that including sprl context in prompts does not improve srl accuracy, and in some cases even degrades performance. however, our experiments revealed a consistency between sprl and srl llm output that mirrors dowty's proto-role theory. in particular, we tested two models, gpt-4o and llama 3 (3b), on synchronous srl and sprl judgments, and found that they both tend to assign proto-agent properties (e.g., instigation, volition) alongside arg0 labels, suggesting that the latent event role encodings of these models align with linguistic theory. additionally, our experiments prompting gpt-4o through a full srl-sprl pipeline provide further evidence that llms are not yet capable of complex, multi-step semantic reasoning needed to annotate sentences for rich event structure @xcite , suggesting that llms deployed in real-world settings may lack the advanced situation-modeling capabilities necessary to make complex decisions. however, our results show that llms do encode something like a latent event role hierarchy, and we suspect that there are better ways to coax the models to employ it. future research should explore, for example, few-shot learning or fine-tuning strategies to better harness llms' event role knowledge. such improvements could not only improve semantic role labeling accuracy, but also deepen our understanding of how llms model event structure: a necessary component in developing robust ai systems with transparent situation-modeling and decision-making capabilities.