P rop B ank Comes of A ge— L arger, Smarter, and more Diverse (2022)

Abstract: This paper describes the evolution of the PropBank approach to semantic role labeling over the last two decades. During this time the PropBank frame files have been expanded to include non-verbal predicates such as adjectives, prepositions and multi-word expressions. The number of domains, genres and languages that have been PropBanked has also expanded greatly, creating an opportunity for much more challenging and robust testing of the generalization capabilities of PropBank semantic role labeling systems. We also describe the substantial effort that has gone into ensuring the consistency and reliability of the various annotated datasets and resources, to better support the training and evaluation of such systems

## introduction twenty years ago traditional statistical machine learning techniques were holding sway and successful stochastic syntactic parsing was on the rise. the availability of accurate syntactic parses opened the door to richer, deeper representations. the second human language technology conference included a presentation on adding predicate argument structure to the penn treebank and the proposition bank (propbank) was born @xcite . over the next few years, with the able guidance of a steering committee consisting of ralph weischedel, mitch marcus, doug appelt, mark villain and ralph grishman, the annotation guidelines and the annotation continued to grow, with the end result of over 110,000 predicate argument structures pointing directly to syntactic nodes in the phrase structure syntax trees of the roughly 50,000 sentences of the penn treebank. the annotation of these structures was guided by a set of approximately 3300 frame files that provided a verb specific set of semantic roles as the arguments for each verb. the substantial size of the data set and the consistency of the annotation gave rise to a flurry of popular semantic role la-beling systems and semantic role labeling shared tasks @xcite that continue to this day. the penn treebank is entirely composed of wall street journal articles, and annotation of additional data taken from the more diverse english genres of the brown corpus allowed for out of domain testing, with predictable dismal results. since that time, darpa and nsf have funded substantial additional propbank annotation, focusing on additional domains and genres for english, as well as additional languages such as chinese, arabic, hindi and urdu. the deep learning revolution has not abated the interest in semantic role labeling performance, and the incorporation of propbank frame files into the abstract meaning representation (amr) editor @xcite , to guide the labeling of the amr nested predicate argument structures, ensures its longevity. this paper details the new genres, domains and datasets that are now available, as well as the expansion of the original propbank verb frame files to adjectival and nominal forms. today propbank has a prominent web presence 1 and plans to evolve and cater to the growing, global, distributed, diverse community by means of a github organization 2 . github supports the infrastructure for streamlining contributions and resolving issues that are bound to arise in the future. multiple versions of stable annotations are made available to the community for promoting open, reproducible research 3 . different versions of the frame lexicon can be viewed and searched online in a human friendly format 4 . we start by reviewing the framework and assumptions for the original propbank and detail 1 http://propbank.org 2 http://github.com/propbank 3 many diverse sources and subcorpora are covered by the sum total of all annotations. access to various data slices is governed by the data and privacy restrictions on the underlying source. a bulk of the data is accessible free of charge for research use upon completion of relevant data use paperwork. the details can be found on the main website. the changes that have been made as it matured in section 2. section 3 describes the additional new domains and genres that are covered in subsequent annotation efforts. in section 4 we provide novel baseline results on these new corpora to stimulate additional research in the robustness and portability of semantic role labeling. finally we summarize our contributions in section 5. ## the proposition bank, then and now propbank @xcite is a paradigm for the development of corpora annotated with predicate argument structures. in its original form, these predicate arguments structures were applied to the syntactic scaffolding provided by the penn treebank. while creating a global inventory of semantic roles was traditionally viewed as too difficult, prop-bank sidestepped the issue by using an "individual thematic roles" approach @xcite in which roles are custom-defined within each (coarsegrained) sense of a predicate. the decision as to what constitutes a semantic role and the use of penn treebank as the syntactic scaffolding for the annotation contributed to high inter-annotator agreement, which led to higher performing machine learning models and fueled interest in the task. propbank has been instrumental in creating a subfield of nlp called semantic role labeling (srl). the following three subsections describe the evolution of propbank in terms of the kind of predicates that were annotated, the changes seen in the data structures as paradigm matured, and the genre of data annotated over the past two decades. ## fresh corpora-new domains and genres several diverse corpora have been propbanked and are now available on our github site. ontonotes the first major expansion to the original wsj propbank was ontonotes foot_4 , described above. bolt the bolt corpus @xcite was treebanked and propbanked as part of the darpa bolt project. it is composed of 628,000 tokens of informal text, divided into sms and text chat data (sms), online discussion forums(df), and translations of informal arabic and chinese data in english (cts). english web text the third corpus, the english web treebank @xcite , is 250k tokens of web text covering weblogs, newsgroups, emails, reviews and online question-answer pairs, and was funded by google. these three corpora not only provide three different genres, but each contains a wide range of subcorpora. one simple illustration of this withincorpus variety can be witnessed in the fact that the conversational speech in ontonotes and in bolt range from 7-10 words per sentence, whereas the ontonotes weblog and bolt discussion forums have an average sentence length of 20 words. one can see that each corpus contains very reduced, conversational examples such as the sms, emails, or the ontonotes telephone conversation data. similarly, each contains long, syntactically complex data-with data such as the bolt discussion forum data differing from traditional newswire, not in complexity, but in editing and syntactic coherence. ## summary and discussion this paper summarized the last twenty years of development and evolution of an approach to semantic role labeling called propbanking. we've outlined the methods for converting propbank to a unified form, and the advantages provided by that unified form and by the larger size of the prop-bank corpora now available. the result is a set of consistently annotated corpora representing diverse genres and domains, all relying on a general set of english frame files. where domain specific frame files are used, they are clearly marked. tools are now available to view the frame files as a whole or as domain-specific subsets on an easily accessible web site. similarly annotated corpora in several other languages are also available. these new datasets offer opportunities for additional testing and evaluation that can advance the ability of srl systems to generalize to new application areas and to new languages. we suggest that testing against the combination of ontonotes, english web treebank, and bolt corpora presented here can provide a more challenging srl evaluation, requiring systems to better handle diverse domains and genres and non-verbal predicates. in the coming year we look forward to toasting both propbank on its 21st birthday and the winning systems of new srl evaluation tasks.