Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities (2023)

Abstract: Although we have witnessed impressive progress in Semantic Role Labeling (SRL), most of the research in the area is carried out assuming that the majority of predicates are verbs. Conversely, predicates can also be expressed using other parts of speech, e.g., nouns and adjectives. However, non-verbal predicates appear in the benchmarks we commonly use to measure progress in SRL less frequently than in some real-world settings – newspaper headlines, dialogues, and tweets, among others. In this paper, we put forward a new PropBank dataset which boasts wide coverage of multiple predicate types. Thanks to it, we demonstrate empirically that standard benchmarks do not provide an accurate picture of the current situation in SRL and that state-of-the-art systems are still incapable of transferring knowledge across different predicate types. Having observed these issues, we also present a novel, manually-annotated challenge set designed to give equal importance to verbal, nominal, and adjectival predicate-argument structures. We use such dataset to investigate whether we can leverage different linguistic resources to promote knowledge transfer. In conclusion, we claim that SRL is far from “solved”, and its integration with other semantic tasks might enable significant improvements in the future, especially for the long tail of non-verbal predicates, thereby facilitating further research on SRL for non-verbal predicates. We release our software and datasets at https://github.com/sapienzanlp/exploring-srl .

## introduction over the years, semantic role labeling @xcite -the task of identifying the semantic relations between predicates and their arguments -has attracted continued interest. enticed by the prospect of acquiring one * equal contribution. notwithstanding the achievements of previous work, we argue that there is still much to be done before the research community can claim srl is even close to being "solved". one of the simplest yet erroneous assumptions about srl is that all predicates -or at least the majority of them -are verbs. quite the contrary, predicates often manifest themselves as nouns, adjectives, and adverbs. for example, in the sentence "sensational robbery at the bank during the night: two suspects on the loose!", the word robbery is a predicate, as it denotes an action, and its arguments are sensational (attribute of the robbery), at the bank (location), during the night (time), and two suspects (agents). we highlight two potential issues in the above example. first, an srl system that analyzes only verbal predicates cannot identify the nominal event in the sentence and, in turn, its semantic constituents. second, nominal events like those expressed in the above sentence are far from rare, being commonly found in several settings, such as newspaper headlines, blog titles, short messages, tweets, and dialogues. perhaps surprisingly, there is limited work on non-verbal predicates, mostly focused on transferring "knowledge" about verbal predicates to nominal ones @xcite . the scarcity of studies on non-verbal predicates might be explained by the way in which current datasets for srl are designed, as they focus primarily on verbal predicates @xcite @xcite . therefore, any progress on non-verbal predicates is often overshadowed by the predominance of verbal instances, resulting in an incomplete picture of the actual situation. the issue is also exacerbated by the fact that, oftentimes, benchmark results are taken at face value. instead, carrying out in-depth analyses is fundamental, as neural networks have been found to learn patterns that are different from those of humans, especially in semantic tasks @xcite . in this paper, we perform a reality check and explore non-verbal predicates in english srl. more specifically, our contributions are as follows: as mentioned above, relying on standard benchmarks does not allow us to properly evaluate the performance of state-of-the-art systems on nonverbal srl. cases in point are the conll shared tasks: conll-2005 covers only verbal predicates; conll-2009 includes verbal and nominal predicates but makes it difficult to compare them, as they belong to two different inventories, propbank and nombank, respectively; conll-2012 and its revision in ontonotes 5.0 (pradhan et al., 2022) do not verbs nouns adjs framesets conll-2009 1090 1337 0 2427 ontonotes 5.0 2215 782 3 2490 pb-examples 5465 1384 1599 7481 pb-unseen 2457 469 1389 4001 in examples also enables a solid evaluation of an srl system on over 4000 predicate senses that are not included in ontonotes 5.0; we call this more challenging testbed pb-unseen. we report statistics on pb-unseen in the last row of cross-type knowledge transfer. now that we have wide-coverage multi-type srl datasets, we can test the ability of srl systems to generalize across types. the main objective of our experiments here is to empirically demonstrate that: i) "knowledge transfer" between predicate types is an unaddressed challenge, and ii) this problem is not apparent in ontonotes, but becomes evident from pb-examples and pb-unseen. to prove these points, we take cn-22 -a state-of-the-art system @xcite -and study its behavior when trained on the entire ontonotes (cn-22 verbs+nouns ), only on its verbal structures (cn-22 verbs ), or only on its nominal structures (cn-22 nouns ). the results on the test set of ontonotes, shown in ## opportunities in the previous section, our experiments show that zero-shot knowledge transfer across predicate types is still challenging. we argue that this problem is caused by two main factors. first, propbank was not designed to aid cross-type knowledge transfer, e.g., the nominal predicate theft.01 is not linked to its verbal equivalent steal.01. second, recent srl systems might have limited capability for recognizing common patterns across different predicate types. we conduct an initial investigation of these aspects and discuss some opportunities for improving non-verbal srl. the role of the linguistic resource. while prop-bank might not be the ideal resource for non-verbal srl, other inventories -based on different linguistic theories -may provide features that could be helpful to aid knowledge transfer between predicate types. after all, previous studies have already shown that language models leverage different hidden layers depending on the linguistic resource used for srl @xcite . here, instead, we take the opportunity to study if there is an inventory whose theoretical principles can aid the generalization capability of an existing srl system on unseen patterns. we thus evaluate empirically the differences between four different inventories, namely, propbank, framenet @xcite , verbnet @xcite , and verbatlas @xcite . 1 to do this, we create parallel-semlink, a multi-inventory benchmark made up of the subset of ontonotes from semlink 2.0 @xcite , whose predicates and arguments are annotated with propbank, framenet, and verbnet. we also include verbatlas annotations thanks to the inter-resource mapping between verbnet, word-net, and verbatlas. 2 for each of these inventories, parallel-semlink includes a training, a validation, and a test set with 7336, 816, and 906 sentences, respectively. while we stress that this experimental setting is severely limited since it assumes that all resources can be mapped to each other 1-to-1, it provides a controlled environment for a fair, direct comparison. to study the impact of the inventory, we evaluate our srl system on each of the linguistic inventories in parallel-semlink (cn-22 propbank , cn-22 framenet , cn-22 verbnet , and cn-22 verbatlas ). the results in challenge-srl. while our multi-inventory semlink-based dataset provides a preliminary indication of the role of a linguistic inventory, it only includes verbal predicates. to further validate the preliminary results obtained on our multi-inventory semlink-based dataset, we create a small challenge test set for verbal, nominal, and adjectival srl, manually annotated with parallel labels for propbank, the most popular inventory, and verbatlas, the most promising inventory (cf. as we can see from leveraging word sense disambiguation. finally, we carry out a preliminary exploration of possible directions that could aid non-verbal srl in the future. while srl research has not dealt with non-verbal semantics, other areas have investigated semantics for different parts of speech, and one of these is word sense disambiguation (wsd). more specifically, wsd is the task of assigning the most appropriate sense to a word in context according to a predefined sense inventory @xcite . it is easy to notice how this task resembles predicate sense disambiguation in srl, the only difference being that wsd is not limited to predicates, as it aims to disambiguate every content word. therefore, we believe that wsd is an interesting candidate to explore whether a different disambiguation task can help to improve the generalization capability of an existing srl system on challenge-srl, i.e., on predicate-argument structures that the srl system did not see at training time. to investigate the effect of wsd on srl, we start by leveraging the fact that verbatlas frames are clusters of wordnet synsets. therefore, we map each synset predicted by amuse-wsd @xcite ), foot_0 a state-of-the-art offthe-shelf wsd system, to a verbatlas frame, and compare them to the prediction of our srl system. most interestingly, if we employ an oracle to pick the best prediction between the wsd baseline and our best srl system, we notice a further improvement (41.5% vs. 26.0%), demonstrating that current state-of-the-art srl systems can still benefit from explicit lexical semantics. we hypothesize that tighter integration of the two tasks may lead to even better improvements in generalization capabilities. ## conclusion and future work in this paper, we carried out a reality check and demonstrated that, despite impressive results on standard benchmarks by state-of-the-art systems, srl is still far from "solved". indeed, thanks to a carefully-designed set of experiments and the introduction of novel, manually-curated, wide-coverage benchmarks, we showed that current srl systems possess inadequate capabilities for transferring knowledge between predicate types. our analyses pointed out that we can address this limitation by working in two directions: leveraging the intrinsic characteristic of frameset resources, including semantics-based clusters and cross-predicate role semantics, and tighter integration of other semantics-based tasks, such as word sense disambiguation, into srl. we hope our work will be a stepping stone for innovative research on high-performance srl systems for non-verbal predicate-argument structures, a problem that still needs extensive investigation. for this reason, we release our software and datasets at https://github.com/sapienzanlp/ exploring-srl .