Structured Tuning for Semantic Role Labeling (2020)

Abstract: Recent neural network-driven semantic role labeling (SRL) systems have shown impressive improvements in F1 scores. These improvements are due to expressive input representations, which, at least at the surface, are orthogonal to knowledge-rich constrained decoding mechanisms that helped linear SRL models. Introducing the benefits of structure to inform neural models presents a methodological challenge. In this paper, we present a structured tuning framework to improve models using softened constraints only at training time. Our framework leverages the expressiveness of neural networks and provides supervision with structured loss components. We start with a strong baseline (RoBERTa) to validate the impact of our approach, and show that our framework outperforms the baseline by learning to comply with declarative constraints. Additionally, our experiments with smaller training sizes show that we can achieve consistent improvements under low-resource scenarios.

## introduction semantic role labeling @xcite is the task of labeling semantic arguments of predicates in sentences to identify who does what to whom. such representations can come in handy in tasks involving text understanding, such as coreference resolution @xcite and reading comprehension (e.g., @xcite . this paper focuses on the question of how knowledge can influence modern semantic role labeling models. linguistic knowledge can help srl models in several ways. for example, syntax can drive feature design (e.g., @xcite @xcite , and can also be embedded into neural network architectures @xcite . in addition to such influences on input representations, knowledge about the nature of semantic roles can inform structured decoding algorithms used to construct the outputs. the srl literature is witness to a rich array of techniques for structured inference, including integer linear programs (e.g., @xcite , bespoke inference algorithms (e.g., @xcite , a* decoding (e.g., @xcite , greedy heuristics (e.g., @xcite , or simple viterbi decoding to ensure that token tags are bio-consistent. by virtue of being constrained by the definition of the task, global inference promises semantically meaningful outputs, and could provide valuable signal when models are being trained. however, beyond viterbi decoding, it may impose prohibitive computational costs, thus ruling out using inference during training. indeed, optimal inference may be intractable, and inference-driven training may require ignoring certain constraints that render inference difficult. while global inference was a mainstay of srl models until recently, today's end-to-end trained neural architectures have shown remarkable successes without needing decoding. these successes can be attributed to the expressive input and internal representations learned by neural networks. the only structured component used with such models, if at all, involves sequential dependencies between labels that admit efficient decoding. in this paper, we ask: can we train neural network models for semantic roles in the presence of general output constraints, without paying the high computational cost of inference? we propose a structured tuning approach that exposes a neural srl model to differentiable constraints during the finetuning step. to do so, we first write the output space constraints as logic rules. next, we relax such statements into differentiable forms that serve as regularizers to inform the model at training time. finally, during inference, our structure-tuned models are free to make their own judgments about labels without any inference algorithms beyond a simple linear sequence decoder. we evaluate our structured tuning on the conll-05 @xcite and conll-12 english srl @xcite shared task datasets, and show that by learning to comply with declarative constraints, trained models can make more consistent and more accurate predictions. we instantiate our framework on top of a strong baseline system based on the roberta @xcite encoder, which by itself performs on par with previous best srl models that are not ensembled. we evaluate the impact of three different types of constraints. our experiments on the conll-05 data show that our constrained models outperform the baseline system by 0.2 f1 on the wsj section and 1.2 f1 on the brown test set. even with the larger and cleaner conll-12 data, our constrained models show improvements without introducing any additional trainable parameters. finally, we also evaluate the effectiveness of our approach on low training data scenarios, and show that constraints can be more impactful when we do not have large training sets. in summary, our contributions are: 1. we present a structured tuning framework for srl which uses soft constraints to improve models without introducing additional trainable parameters. foot_0 2. our framework outperforms strong baseline systems, and shows especially large improvements in low data regimes. ## model & constraints in this section, we will introduce our structured tuning framework for semantic role labeling. in §2.1, we will briefly cover the baseline system. to that, we will add three constraints, all treated as combinatorial constraints requiring inference algorithms in past work: unique core roles in §2.3, exclusively overlapping roles in §2.4, and frame core roles in §2.5. for each constraint, we will discuss how to use its softened version during training. we should point out that the specific constraints chosen serve as a proof-of-concept for the general methodology of tuning with declarative knowledge. for simplicity, for all our experiments, we use the ground truth predicates and their senses. ## experiments & results in this section, we study the question: in what scenarios can we inform an end-to-end trained neural model with declarative knowledge? to this end, we experiment with the conll-05 and conll-12 datasets, using standard splits and the official evaluation script for measuring performance. to empirically verify our framework in various data regimes, we consider scenarios ranging from where only limited training data is available, to ones where large amounts of clean data are available. ## ablations & analysis in §3, we saw that constraints not just improve model performance, but also make outputs more structurally consistent. in this section, we will show the results of an ablation study that adds one constraint at a time. then, we will examine the sources of improved f-score by looking at individual labels, and also the effect of the top-k relaxation for the constraint o. furthermore, we will examine the robustness of our method against randomness involved during training. we will end this section with a discussion about the ability of constrained neural models to handle structured outputs. ## discussion & conclusion semantic role labeling & constraints the srl task is inherently knowledge rich; the outputs are defined in terms of an external ontology of frames. the work presented here can be gener-alized to several different flavors of the task, and indeed, constraints could be used to model the interplay between them. for example, we could revisit the analysis of @xcite , who showed that the propbank a2 label takes on multiple meanings, but by mapping them to verbnet, they can be disambiguated. such mappings naturally define constraints that link semantic ontologies. constraints have long been a cornerstone in the srl models. several early linear models for srl (e.g. @xcite @xcite modeled inference for propbank srl using integer linear programming. @xcite used markov logic networks to learn and predict semantic roles with declarative constraints. the work of @xcite showed that certain srl constraints admit efficient decoding, leading to a neural model that used this framework @xcite . learning with constraints has also been widely adopted in semisupervised srl (e.g., @xcite . with the increasing influence of neural networks in nlp, however, the role of declarative constraints seem to have decreased in favor of fully end-toend training (e.g., @xcite . in this paper, we show that even in the world of neural networks with contextual embeddings, there is still room for systematically introducing knowledge in the form of constraints, without sacrificing the benefits of end-to-end learning. @xcite and @xcite developed models for structured learning with declarative constraints. our work is in the same spirit of training models that attempts to maintain output consistency.