Probing for Predicate Argument Structures in Pretrained Language Models (2022)

Abstract: Thanks to the effectiveness and wide availability of modern pretrained language models (PLMs), recently proposed approaches have achieved remarkable results in dependency- and span-based, multilingual and cross-lingual Semantic Role Labeling (SRL). These results have prompted researchers to investigate the inner workings of modern PLMs with the aim of understanding how, where, and to what extent they encode information about SRL. In this paper, we follow this line of research and probe for predicate argument structures in PLMs. Our study shows that PLMs do encode semantic structures directly into the contextualized representation of a predicate, and also provides insights into the correlation between predicate senses and their structures, the degree of transferability between nominal and verbal structures, and how such structures are encoded across languages. Finally, we look at the practical implications of such insights and demonstrate the benefits of embedding predicate argument structure information into an SRL model.

## introduction semantic role labeling (srl) is often defined informally as the task of automatically answering the question "who did what to whom, where, when and how?" @xcite and is, therefore, thought to be a fundamental step towards natural language understanding @xcite . over the past few years, srl has started to gain renewed traction, thanks mainly to the effectiveness and wide availability of modern pretrained language models (plms), such as elmo @xcite , bert @xcite and bart @xcite . current approaches have, indeed, attained impressive results on standard evaluation benchmarks for dependency-and span-based, multilingual and cross-lingual srl @xcite @xcite @xcite . despite the remarkable benefits provided by the rich contextualized word representations coming from plms, the novelties introduced in recent stateof-the-art models for srl revolve primarily around developing complexities on top of such word representations, rather than investigating what happens inside a plm. for example, the srl systems of @xcite and @xcite take advantage only of bert's uppermost hidden layers to build their input word representations. however, the revolution that plms have sparked in numerous areas of natural language processing (nlp) has motivated researchers in the community to investigate the inner workings of such models, with the aim of understanding how, where, and to what extent they encode information about specific tasks. this research has revealed that different layers encode significantly different features @xcite . in perhaps one of the most notable studies in this direction, @xcite demonstrated empirically that bert "rediscovers" the classical nlp pipeline, highlighting that the lower layers tend to encode mostly lexicallevel information while upper layers seem to favor sentence-level information. although recent analyses have already provided important insights into which layers of a plm are more relevant for srl and how their relative importance is affected by the linguistic formalism of choice @xcite , not only do these analyses treat srl as an atomic task but they also do not explore taking advantage of their insights to improve current state-of-the-art srl systems. indeed, the srl pipeline is usually divided into four main steps: predicate identification and disambiguation, and argument identification and classification. to address this gap, in this paper we therefore take an in-depth look at how predicate senses and their predicate argument struc-tures (pass) are encoded across different layers of different plms. on the one hand, we provide new insights into the capability of these models to capture complex linguistic features, while on the other, we show the benefits of embedding such features into srl systems to improve their performance. our contributions can be summarized as follows: we hope that our work will contribute both to the understanding of the inner workings of modern pretrained language models and to the development of more effective srl systems. we release our software for research purposes at https://github . ## related work probing pretrained language models. the unprecedented capability of modern plms to provide rich contextualized input representations took the nlp community by storm. alongside the rising wave of successes collected by plms in an ever increasing number of areas, researchers started to question and investigate what happens inside these models and what they really capture, probing for knowledge and linguistic properties @xcite @xcite . this body of work quickly attracted increasing attention and grew to become a field of study with a name of its own: bertology @xcite . probing a plm usually consists in defining a very precise task (e.g., identifying whether two words are linked by a syntactic or semantic relation), and then in designing and training a simple model, called a probe, to solve the task using the contextualized representations provided by the plm. the idea is to design a probe that is as simple as possible, often consisting of a single-layer model: if the probe is able to address the task, then it must be thanks to the contextual information captured by the plm as the expressiveness of the probe itself is limited by its simplicity. one could argue that some complex relations may require a non-linear probe @xcite which can reveal hidden information as long as it is accompanied by control experiments @xcite to verify that it is still extracting information from the underlying plm, rather than merely learning to solve the probing task. over the past few years, these probing techniques have been used to great effect and revealed that plms have been "rediscovering" the classical nlp pipeline @xcite , and that they often encode distances between syntactic constituents @xcite , lexical relations @xcite and morphology @xcite , inter alia. probing techniques for srl. as in several other fields of nlp, recent studies have aimed to shed some light on how, where and to what extent plms encode information relevant to srl. among others, @xcite devised an edge probing mechanism aimed at ascertaining the capability of bert to identify which semantic role ties a given predicate to a given argument span, and showed that this task is "solved" mainly by the middle layers of bert. toshniwal et al. (2020) proposed and compared several techniques for better combining the contextualized representations of a plm, finding that applying max pooling or performing a weighted average are two robust strategies for srl. more recently, kuznetsov and gurevych (2020) designed a probe to analyze how different linguistic ontologies -essential to the task in that they define predicate senses and semantic roles explicitly -require features that are encoded at different layers of a plm. in this paper, we follow the line of research laid out by the afore-mentioned work, probing plms with the objective of understanding where and to what extent they encode a predicate argument structure into the contextualized representation of a predicate. recent advances in srl. thanks to their effectiveness, plms are now the de facto input representation method in srl @xcite @xcite . recently proposed approaches have achieved impressive results on several gold benchmarks @xcite , both in span-based and in dependency-based srl, but also in multilingual and cross-lingual srl, even though there still seems to be a significant margin for improvement in out-of-domain settings. the innovations put forward by such approaches, however, have mainly focused on architectural novelties built on top of plms: cai et al. (2018) proposed the first end-to-end architecture; he et al. (2019) and cai and lapata (2019) successfully exploited syntax in multilingual srl; marcheggiani and titov (2020) took advantage of gcns to capture distant semantic relations; conia and navigli (2020) devised a language-agnostic approach to bridge the gap in multilingual srl; blloshmi et al. (2021) and paolini et al. (2021) tackled the task as a sequence generation problem; conia et al. (2021) introduced a model to perform cross-lingual srl across heterogeneous linguistic inventories. however, if we look back at past work, it is easy to realize that we lack a study that provides an indepth look into plms and a hint at how to better exploit them in future srl systems. ## probing for predicate senses and their predicate-argument structures as mentioned above, some studies have already investigated how semantic knowledge is distributed among the inner layers of current plms, finding that information useful for srl is mainly stored in their middle layers @xcite . however, such studies have considered srl as an atomic task, while instead the srl pipeline can be thought of as being composed of four different subtasks: for our study, it is important to note that, in many popular ontologies for srl, predicate senses or frames are often tightly coupled to their possible semantic roles. in other words, the set of possible semantic roles that can be linked to a predicate p is defined according to the sense or frame of p. hereafter, given a predicate p, we refer to its set of possible semantic roles as the roleset of p. for example, the predicate love as in "he loved everything about her" belongs to the framenet @xcite frame experiencer_focused_emotion which defines a roleset composed of {experiencer, content, . . ., degree}. the same predicate sense has different rolesets in other ontologies, for example {arg0 (lover), arg1 (loved)} in the english propbank @xcite and {experiencer, stimulus, . . ., cause} in verbatlas @xcite . ## integrating predicate-argument structure knowledge now that we have provided an in-depth look at how sense and roleset information is encoded at different inner layers of current plms (section 3.2), highlighted the differences in how plms encode verbal and nominal predicates (section 3.3), and revealed that multilingual plms capture semantic knowledge at similar layers across two diverse languages (section 3.4), one may wonder how we can take advantage in a practical setting of what we have learned so far. in this section, we study how we can improve a modern system for end-to-end srl by integrating sense and roleset knowledge into its architecture. ## conclusion in this paper, we probed plms for pass: differently from past work, we dissected srl into its core subtasks and analysed how plms encode predicate-argument structure information such as predicate senses and their rolesets. in our analysis, we observed that, despite the intrinsic connection between predicate senses and their rolesets that exists in several popular srl inventories, different plms encode their features across significantly different layers. what is more, we also discovered that verbal and nominal predicates and their pass are represented differently, making verbalto-nominal srl transfer far from trivial, and providing an empirical explanation for why previous attempts in this direction have struggled to obtain strong results. furthermore, our analysis revealed that current multilingual language models encode pass similarly across two very different languages, namely, english and chinese. finally, in contrast to previous work on probing, we put together what we learned and demonstrated a practical application of our findings by devising simple yet effective techniques for the integration of predicate-argument structure knowledge into a state-of-the-art end-to-end architecture for srl.