P ri M e SRL -Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation (2023)

Abstract: Semantic role labeling (SRL) identifies the predicate-argument structure in a sentence. This task is usually accomplished in four steps: predicate identification, predicate sense disambiguation, argument identification, and argument classification. Errors introduced at one step propagate to later steps. Unfortunately, the existing SRL evaluation scripts do not consider the full effect of this error propagation aspect. They either evaluate arguments independent of predicate sense (CoNLL09) or do not evaluate predicate sense at all (CoNLL05), yielding an inaccurate SRL model performance on the argument classification task. In this paper, we address key practical issues with existing evaluation scripts and propose a more strict SRL evaluation metric PriMeSRL. We observe that by employing PriMeSRL, the quality evaluation of all SoTA SRL models drops significantly, and their relative rankings also change. We also show that PriMeSRLsuccessfully penalizes actual failures in SoTA SRL models.

## introduction semantic role labeling (srl) extracts predicateargument structures from a sentence, where predicates represent relations (verbs, adjectives, or nouns) and arguments are the spans attached to the predicate demonstrating "who did what to whom, when, where, and how." as one of the fundamental natural language processing (nlp) tasks, srl has been shown to help a wide range of nlp downstream applications such as natural language inference @xcite , question answering @xcite @xcite , machine translation @xcite , content moderation and verification @xcite , information extraction @xcite ; zhang * b work done while at ibm research [derick] broke the [window] with a [hammer] to [escape] . a0 break.01 a1 a2 am-prr nsubj det obj mark det obl mark obl root figure 1: an srl example with head-based semantic roles on top of universal dependencies annotation. given a sentence, a typical srl system obtains predicate-argument structure by following a series of four steps: 1) predicate identification; 2) predicate sense disambiguation; 3) argument identification; and 4) argument classification. the predicate senses and their argument labels are taken from inventories of frame definitions such as proposition bank (propbank) @xcite , framenet @xcite , and verb-net @xcite . the accuracy of srl extraction is affected by the correctness of each of these steps. consider the example in figure 1 using propbank foot_0 annotations: the srl system must: finally, this example has one additional modifier: the am-prp (the purpose). figure 1 illustrates the same analysis on top of the universal dependencies annotations where only the tokens' head of phrases are annotated with the proper argument. to obtain a completely correct predicateargument structure both the predicate sense and all of its associated arguments need to be correctly extracted. mistakes introduced at one step may propagate to later steps, leading to further errors. for instance, in the above example, a wrong predicate sense 'break.02' (break in or gain entry) has not only a different meaning from 'break.01' (break) but also a different set of arguments. in many cases, even if an argument for a wrong predicate sense is labeled with the same numerical roles (a1, a2, etc), their meanings can be very different. therefore, in general, the labels for argument roles should be considered to be incorrect when the predicate sense itself is incorrect. however, existing srl evaluation metrics (e.g. @xcite ) do not penalize argument labels in such cases. the currently used evaluation metrics also do not evaluate discontinuous arguments accurately. some arguments in the propbank original corpora have discontinuous spans that all refer to the same argument. this can happen for a number of reasons such as in verb-particle constructions. in a dependency-based analysis, these arguments end up being attached to distinct syntactic heads @xcite . take as an example the sentence, "i know your answer will be that those people should be allowed to live where they please as long as they pay their full locational costs." for the predicate "allow.01," the a1 (action allowed) is the discontinuous span "those people" (a1) and "to live where they please as long as they pay their full locational costs" (c-a1). existing evaluation metrics treat these as two independent labels. a similar problem exists for the evaluation of reference arguments (r-x). for example, in the sentence "this is exactly a road that leads nowhere", for the predicate "lead.01", the a0 "road" is referenced by c-a0 "that". if a0 is not correctly identified, the reference c-a0 is meaningless. in this paper, we conduct a systematic analysis of the pros and cons of different evaluation metrics for srl, including: we then propose a new metric for evaluating srl systems in a more accurate and intuitive manner in section 3, and compare it with currently used methods in section 4. primesrl is available at https: //github.com/universalpropositions/primesrl-eval. ## existing evaluation metrics for srl most of the existing evaluation metrics came from shared tasks for the development of systems capable of extracting predicates and arguments from natural language sentences. in this section, we summarize the approaches to srl evaluation in the shared tasks from semeval and conll. ## the proposed approach we propose primesrl, a new metric for evaluating srl systems, based on the following high-level rules that aim to overcome the drawbacks in existing metrics: ## comparisons with existing metrics in this section, we discuss the effectiveness of existing srl evaluation metrics and demonstrate how primesrl differs in various use cases, using sota neural srl models as test models. ## conclusion in this paper, we highlighted key issues with existing srl evaluation metrics and showed that the proposed evaluation metric, primesrl, scores sota srl models in a more accurate and intuitive manner. by releasing our evaluation code, we plan to promote these metrics in the community in order to improve the evaluation quality for srl systems that contribute to downstream applications.