Label Definitions Improve Semantic Role Labeling (2022)

Abstract: Argument classification is at the core of Semantic Role Labeling. Given a sentence and the predicate, a semantic role label is assigned to each argument of the predicate. While semantic roles come with meaningful definitions, existing work has treated them as symbolic. Learning symbolic labels usually requires ample training data, which is frequently unavailable due to the cost of annotation. We instead propose to retrieve and leverage the definitions of these labels from the annotation guidelines. For example, the verb predicate “work” has arguments defined as “worker”, “job”, “employer”, etc. Our model achieves state-of-the-art performance on the CoNLL09 dataset injected with label definitions given the predicate senses. The performance improvement is even more pronounced in low-resource settings when training data is scarce.

## introduction semantic role labeling (srl) is an essential nlp task of answering the question of "who did what to whom, when, where and how." formally, a semantic role is assigned to each argument of a predicate in a sentence. srl has been shown to help a wide range of nlp applications such as natural language inference @xcite , question answering @xcite @xcite and machine translation @xcite . it can also be used as a pre-processing step for tasks such as information extraction @xcite . learning from ample labeled examples is the predominant paradigm in many nlp tasks @xcite , including srl. however, labeled data is costly and often lacking in many tasks, domains, and languages. one attempt at this issue, made possible by recent advancement of language models, is to inject task descriptions into the data so that models become "aware" of the task requirements and the meaning of the labels. this technique has successfully been used in sentiment analysis @xcite , event extraction @xcite , intent detection @xcite , word sense disambiguation @xcite , and many other tasks @xcite . in srl, while the label space is particularly sparse as each predicate has different semantic roles, definitions are readily available for all possible arguments of supported predicates. while early work has used label definitions for frame generalization specific to framenet @xcite @xcite @xcite , there has been no work that targets general srl in such a label-aware fashion. in srl, the semantic roles are defined specifically for each predicate sense. in figure 1 , given the predicate "work" and its sense, the definitions of its arguments can be found in frames provided by corpora such as propbank @xcite . while previous work has treated argument labels as symbolic, we propose to inject their textual descriptions into the data, with the hypothesis that pre-trained language models can leverage them, analogous to answering questions like "what is the employer of 'work' in the sentence". we show that injecting textual descriptions helps language models (1) to outperform the previous state-of-the-art by more than 1 f1 on the conll09 out-domain test set, (2) to improve the model's ability to generalize to unseen or low-frequency predicates, and the conll 2009 shared task @xcite proposed a dependency-based srl dataset, henceforth referred to as conll09, where arguments are represented as head words instead of spans. it is one of the most commonly used srl training dataset and benchmark (statistics shown in the argument classification (ac) task is to label each token in a sentence as either non-argument in-domain out-domain @xcite 96.9 90.6 (ours) roberta-base 96.7 88.5 or otherwise a semantic role, given a predicate. the task is thus formulated as token classification. we enclose the predicate with separator tokens. an example is shown in the topmost of figure 1 . as before, using a simple roberta-base model, we achieve performance on par with the current stateof-the-art ( ## argument label definitions we propose to expand argument classification data by injecting argument label definitions (acd) with semantic meanings to the arguments, which are readily available. our approach does not focus on and is agnostic to model architecture. ## experiments and results we experiment with 2 settings, all using the roberta-base model mentioned before. first, we consider a model trained and tested on the ordinary ac data and another on the acd data. for the acd model, we report performance both using gold predicate senses and using predicted senses in test time. each model uses the default hyperparameters from huggingface transformers @xcite model performances are shown in on the out-domain test set, our acd model with gold sense outperforms current state-of-the-art by 1.1 argument f1, and the ac model by 1.6. with predicted sense of accuracy of 88.5% (see ## low-resource settings we show that inductive biases such as label definitions benefit performance the most when training data is scarce. in srl, such scenario is common in domains with jargon and in applications that require srl with minimal supervision. low-frequency predicates. previous work has found that srl suffers from the long-tail phenomenon, where most predicates are rare words @xcite . we experiment with disjoint subsets of the test data with predicate senses of different frequencies. in ## conclusion and future work we show that definitions of arguments advances state-of-the-art of semantic role labeling on conll09, and even more notably in low-resource settings. the observed performance gap between acd with gold and predicted sense suggests that a more competent psd model is needed. future work may also expand our approach to span-based srl datasets, or multilingual settings. our ac and acd models are implemented as a robertafortokenclassification object 10 . details of configurations and hyperparameters can be found in their documentations. we run all experiments on nvidia v100 gpus with 16g memory. during training, we save the model with the best argument f1 on the development set each 100 training steps. we run each experiment (e.g. training acd on conll09) for up to 48 hours, or when the best model is not updated for 5,000 training steps, whichever is sooner, before evaluating the best model on the test set.