Joint End-to-end Semantic Proto-role Labeling (2023)

Abstract: Semantic proto-role labeling (SPRL) assigns properties to arguments based on a series of binary labels. While multiple studies have evaluated various approaches to SPRL, it has only been studied in-depth as a standalone task using gold predicate/argument pairs. How do SPRL systems perform as part of an information extraction pipeline? We model SPRL jointly with predicate-argument extraction using a deep transformer model. We find that proto-role labeling is surprisingly robust in this setting, with only a small decrease when using predicted arguments. We include a detailed analysis of each component of the joint system, and an error analysis to understand correlations in errors between system stages. Finally, we study the effects of annotation errors on SPRL.

## introduction semantic analyses of text have been framed @xcite as extracting structured information in the form of predicates, arguments, and their relations, often called semantic roles. multiple schemas have been proposed for structuring semantic roles, each with its own benefits and challenges. semantic proto-roles @xcite offer a way to decompose traditional inventories of thematic roles into simple properties that are both easier to annotate and more generalizable to unseen arguments. these emerge from dowty's protorole theory, which assigns properties to arguments based on how agent-like (volition, sentience) or patient-like (change of state, was used) they are. for example, in the sentence "the boy threw a rock," categorical role inventories assign argument "boy" the role agent, and argument "rock" the role patient. work on decompositional semantics 1 has formulated the task of semantic proto-role labeling as the assignment of 14 different binary properties to arguments @xcite . multiple systems have been proposed for automatically assigning proto-roles to predicatearguments pairs in text @xcite @xcite , which have established the feasibility and best practices for semantic proto-role labeling (sprl). at the same time, this task continues to be either treated in total isolation, assuming gold predicates and arguments, or included in universal dependency semantics (uds) parsing pipelines @xcite , which has not included fine-grained analysis on semantic protorole properties themselves. how well does sprl work when integrated into a semantic extraction pipeline? are earlier errors compounded by sprl? are the same tokens challenging for each stage of the pipeline? we answer these questions by constructing a joint multi-task model for identifying predicates and arguments, and assigning proto-role properties. competitive with state-of-the-art for both dependency-and span-based sprl evaluation, a careful component-wise analysis of our system allows us to make the following contributions. 1) despite sprl labeling errorful predicate argument predictions, our results are still competitive with having gold predicates and arguments, and far surpass the only previous work that predicts protoroles jointly with predicates and arguments. future work should include sprl scores with predicted arguments. 2) errors in predicates and arguments do not negatively affect sprl because the same tokens that are challenging for argument identification are challenging for sprl. 3) we find that most sprl errors come from arguments with annotator disagreement, which suggests that these are inherently hard; removing unskilled annotators doesn't change performance, suggesting that conflict alone is not the source of the problem. we discuss implications for future work on sprl after our analysis. ## semantic proto-roles semantic role labeling (srl) was first formulated as a natural language understanding task by @xcite and quickly proliferated @xcite @xcite as resources and common evaluation frameworks were introduced @xcite @xcite @xcite . srl assigns relationships or roles of an argument and its predicate. various labels build on different linguistic theories: label inventories that are small and coarse versus large and fine-grained. common roles include agent, patient, goal, and location. neuralbased systems that assign srl jointly with other related tasks, such as predicate and argument identification, perform just as well or better than models trained on only srl @xcite @xcite @xcite . argument identification can be formulated as dependency-based (find the argument's syntactic head) or span-based (find the entire argument span). the conll 2004 and 2005 shared tasks @xcite used spans: an argument is correct only if all argument tokens are correctly identified with the correct argument role. conll 2008 and 2009 @xcite used a dependency-based method, which only requires that the syntactic head of the argument be tagged with the correct argument role. understandably, span-based is more challenging and scores lag behind dependency-based systems @xcite . sprl @xcite offers an alternative @xcite ) by decomposing traditional semantic roles into properties. the two protoroles are "cluster-concepts" called proto-agent and proto-patient, which each correspond to an inventory of properties. certain properties (such as volition or instigation) tend to belong to proto-agents, while others (such as change of state and was used) tend to belong to proto-patients. this analysis offers increased granularity but without sparsification of the training data. the state-of-the-art sprl dependency-based system fine-tunes bert @xcite with a multi-layer perceptron to assign labels using a linear combination of different bert layer embeddings @xcite . for span-based, the leading system uses an attention-based ensemble and trainable "argument marker embeddings" to indicate which tokens are arguments @xcite . @xcite and @xcite jointly predict uds graph structures (i.e., the spans of predicates and arguments) with all uds properties, including semantic proto-roles. both use a sequence-to-graph transductive model, and @xcite is able to improve the transductive model by integrating transformer architecture. systems are rarely evaluated in both dependency-and span-based settings, and none have been evaluated on anything but gold predicates and arguments. ## data we report results on two english-language datasets for sprl: spr1 @xcite and spr2 @xcite . spr1 contains 4,912 wall street journal sentences from propbank (kingsbury @xcite @xcite annotated by a single annotator based on a set of 18 proto-role properties. 9,738 arguments were annotated for the likelihood (on a likert scale from 1 to 5) that a property holds for that argument. spr2 contains 2,758 english web treebank @xcite sentences annotated for a smaller set of 14 properties using a revised, streamlined protocol. in this release, multiple annotators ensured two-way redundancy for each property judgment. following previous work @xcite @xcite , we formulate sprl as a 18 (spr1) or 14 (spr2) way multi-label binary classification problem and map likert labels {1, 2, 3} to 0, and {4, 5} to 1. the task has also been formulated as a regression problem in which sprl scores are predicted as continuous values @xcite ), but we do not include this formulation as a part of our analysis. we additionally map judgments labeled "inapplicable" to 0 to ensure consistency with previous work. we use standard train/dev/tests splits provided in the data. we additionally do analysis on inter-annotator agreement in spr2 shown in appendix b.2. ## joint end-to-end sprl we construct a joint end-to-end sprl system based on bert @xcite with classification heads for each sub-task (figure 1 ). we fine-tune bert and the encoder parameters are shared across tasks. we favor bert as opposed to newer encoders for a direct comparison to @xcite and @xcite . 2 we construct representations of the input sentences to enable the system to efficiently identify predicates, arguments, and sprl. for each sentence, we construct an instance with a candidate predicate prepended onto the sentence with a separator. we use linear classification heads foot_1 with sigmoid functions to produce classification probabilities for each token. we place a classification head on the prepended predicate token to determine if it is a predicate. for argument and sprl, we use separate classification heads on each token in the sentence. dependency-based models predict argument for the dependency head only, whereas span-based models use an ioe tagging and softmax outputs. the resulting argument-tagged subwords are pooled and concatenated with the predicate representation taken from the sentence into the sprl binary classification heads. since we create an input for each possible predicate, we reduce the number of examples originating from incorrect predicates by only using predicate candidates that were verbs. full model and training details appear in appendix a.3. ## experiments we run multiple experiments to isolate the behavior of different components of our system, such as training on only sprl, as well as the full pipeline. for all experiments, we train both a dependencybased and span-based model. spr1 spr2 recall strict f1 recall strict f1 dependency prediction preds arg heads properties preds arg heads properties gold predicates -93.2 77.5 (-8) -95.7 79.0 (-3.4) + predicate prediction 94.8 95.2 78.0 (-7.3) 83.4 97.8 81.8 (-2) recall strict f1 recall strict f1 span prediction preds arg spans properties preds arg spans properties gold predicates -91.6 78.8 (-5.3) -86.7 77.8 (-4.1) + predicate prediction 95.2 91.2 77.6 (-6.7) 92.4 87.6 74.7 (-6.9) scoring sprl is typically reported as micro/macro averaged f1 across the individual spr binary properties. we report gold f1 scores that assume the previous stages of the pipeline produced correct predicates and arguments. however, when considering sprl run on predicted predicates and arguments, we need to adjust the scoring such that we penalize the sprl score due to mistakes earlier in the pipeline. for other tasks, such as entity linking, we can simply mark a link as "missed" if we fail to recognize an entity with a ner system. however, because sprl is a binary classification task, scoring is more complex. we consider two different sprl scoring methods for false negative predicate or arguments: (1) a lenient score that assumes 0 for all properties, which means that missed arguments do not have any of the properties. (2) a strict score that forces the label incorrect for all properties, which assumes we get all property predictions wrong thereby marking them incorrect. we do not modify the sprl score for false positive arguments (for which there are no gold labels) since this would change the set of arguments over which each run of the system is evaluated. for example, in the sentence "bob sat on the chair and i laid on the ground", if the model predicts that "ground" is an argument for "sat", then the model would produce property predictions for "ground" even though there are no annotations for this token. those predictions are ignored entirely, because it is not guaranteed that other runs will also include predictions for this token. we evaluate each component of our system separately to determine the effects of the pipeline. (1) train only the sprl classifiers using gold predicates and arguments. (2) train arguments and sprl assuming gold predicates. (3) train predicates, arguments, and sprl using inputs first filtered to consider only verbs as predicates. we repli-cate this training for both span and dependencybased predictions. for each setting, we evaluate under different conditions by decoding assuming gold or predicted labels from earlier in the pipeline. ## results we present an overview of the results for our joint system, but full results appear in appendix b. how are sprl errors related to mistakes earlier in the pipeline? sprl performance was much lower for arguments that would have been missed earlier in the pipeline. (see ## discussion our end-to-end sprl system demonstrates the efficacy of sprl when combined with a full system. we are competitive with both span-based and dependency-based models and find that joint identification of predicates and arguments still produces a high-performing sprl system. future work should evaluate this setting, using both spanand dependency-based models and our proposed scoring method. furthermore, our work points to the need for focused improvement on challenging arguments, which is harming both argument identification and sprl. do these errors show the limits of sprl since annotators also get them wrong? do we need better encoder training? will downstream tasks that consume sprl labels be robust to these errors? what is the feasibility of a reinforcement learning system that trains on the model's own output? these questions remain for future work.