Predicate Representations and Polysemy in V erb N et Semantic Parsing (2021)

Abstract: Despite recent advances in semantic role labeling propelled by pre-trained text encoders like BERT, performance lags behind when applied to predicates observed infrequently during training or to sentences in new domains. In this work, we investigate how role labeling performance on low-frequency predicates and out-of-domain data can be further improved by using VerbNet, a verb lexicon that groups verbs into hierarchical classes based on shared syntactic and semantic behavior and defines semantic representations describing relations between arguments. We find that VerbNet classes provide an effective level of abstraction, improving generalization on low-frequency predicates by allowing them to learn from the training examples of other predicates belonging to the same class. We also find that joint training of VerbNet role labeling and predicate disambiguation of VerbNet classes for polysemous verbs leads to improvements in both tasks, naturally supporting the extraction of VerbNet’s semantic representations.

## introduction semantic role labeling (srl) is a form of shallow semantic parsing that involves the extraction of predicate arguments and their assignment to consistent roles with respect to the predicate, facilitating the labeling of e.g. who did what to whom @xcite . srl systems have been broadly applied to applications such as question answering @xcite , machine translation @xcite , dialog systems @xcite , metaphor detection @xcite , and clinical information extraction @xcite . recent approaches to srl have achieved billy consoled the puppy pb arg0 console.01 arg1 vn stimulus amuse-31.1 experiencer billy walked the puppy pb arg0 walk.01 arg1 vn agent run-51.3.2-2-1 theme the assignment of role labels to a predicate's arguments is dependent upon the predicate's sense. propbank @xcite divides each predicate into one or more rolesets, which are coarsegrained sense distinctions that each provide a set of core numbered arguments (a0-a5) and their corresponding definitions. verbnet (vn) groups verbs into hierarchical classes, each class defining a set of valid syntactic frames that define a direct correspondence between thematic roles and syntactic realizations, e.g. agent rel patient (e.g. john broke the vase) or patient rel (e.g. the vase broke) for break-45.1 @xcite . recent propbank (pb) semantic role labeling models have largely eschewed explicit predicate disambiguation in favor of direct prediction of semantic roles in end-to-end trainable models @xcite @xcite . this is possible for several reasons: first, prop-bank's core roles and modifiers are shared across all predicates, allowing a single classifier to be trained over tokens or spans. second, although definitions of pb roles are specific to the different senses of each predicate, efforts are made when creating rolesets to ensure that a0 and a1 exhibit properties of @xcite . finally, pb rolesets are defined based on vn class membership, with predicates in the same classes thus being assigned relatively consistent role definitions @xcite . unlike propbank, verbnet's thematic roles are shared across predicates and classes with consistent definitions. however, vn roles are more dependent on the identity of the predicate @xcite . examples of propbank and verbnet roles illustrating this are given in in this work, we evaluate alternative approaches for incorporating verbnet classes in english verb-net and propbank role labeling. we propose a joint model for srl and vn predicate disambiguation (vn classification), finding that joint training leads to improvements in vn classification and role labeling for out-of-domain predicates. we also evaluate vn classes as predicate-specific features. using gold classes, we observe significant improvements in both pb and vn srl. we also observe improvements in vn role labeling when using predicted classes and features that incorporate all valid classes for each predicate 1 . ## background and related work verbnet verbnet is a broad-coverage lexicon that groups verbs into hierarchical classes based on shared syntactic and semantic behavior @xcite . each vn class is assigned a set of thematic 1 our code is available at https://github.com/ jgung/verbnet-parsing-iwcs-2021 . ## verbnet classes as predicate features verbs belonging to the same vn class share syntactic and semantic properties and the same set of thematic roles and syntactic frames. replacing a predicate in a sentence with a different verb from the same class typically produces a syntactically coherent sentence and does not impact the proposition's thematic role labels. vn classes may thus provide an effective level of abstraction for predicates in srl. we hypothesize that using vn classes as predicate-specific features may help reduce sparsity issues for low-frequency and out-of-vocabulary (oov) verbs. intuitively, training examples for each member verb within a class contribute to the estimation of parameters associated with all other members of the same class, enabling the fine-tuning of predicate-level features even for oov predicates. for example, a verb like traipse may rarely or never occur during training, but may belong to a class which appears hundreds of times in the form of more common verbs like run or rush. we investigate whether by sharing parameter updates across vn members, we can further improve generalization on infrequent verbs. methodology intuitively, bert's nsp pretraining task encourages some level of focus on sent. b tokens from attention heads when processing tokens in sent. a. the predicate feature presented by @xcite and applied in our baseline model uses the predicate token as the sent. b input to bert and thus allows the encodings of tokens in a sentence to be conditioned directly on the predicate. we propose to include tokens corresponding to the predicate's vn class as additional features as part of sent. b. to realize this, we concatenate the corresponding vn class id to sent. b along with the predicate, updating the inputs given in equation 1 : verbnet classification vn classes can be predicted automatically using a word sense disambiguation system. we propose a simple model for verbnet classification: fine tune a pre-trained bert encoder by applying a feedforward multilayer perceptron (mlp) classifier over all vn classes to the bert encoding associated with the first wordpiece of the target predicate. we again condition bert on the target predicate by including it as a feature (w p ) in sent. b: this single classifier formulation is possible for lexicons like vn and framenet in which predicates share senses from a global sense inventory. while individual predicates have a specific set of valid senses, their senses are shared from the global lexicon. @xcite demonstrate that a single classifier approach to vn classification achieves competitive performance when using shared semantic features. intuitively, by training the classifier across multiple verbs, the model parameters specific to each sense receive more updates, with infrequent verb-class pairs also benefiting from the examples of other verbs within the same class. at inference time, we constrain sense predictions to predicate-sense combinations observed in the training data, selecting the highestscoring valid sense given the predicate. we evaluate models using both predicted and gold (ground truth) classes for w s as predicted class and gold class respectively. ## joint verbnet classification and srl features that are useful for srl may also be useful in predicting the sense of a predicate. for example, surface-level syntactic awareness that the argument of a predicate is a clause instead of a noun phrase may change the expected sense of a verb (bring-11.3 vs. characterize-29.2): bob took mary to the doctor. john took mary to be a doctor. the semantic classes of arguments are also often important in determining the sense of a given predicate (dub-29.3.2 vs. get-13.5.1): john called mary a name. john called mary a car. this dependency between srl and predicate sense disambiguation together with the prevalence of shared features between the two tasks makes them a good candidate for multi-task learning @xcite . ## experiments all models are implemented using tensorflow 1.13 @xcite and are trained on a single nvidia gtx 1080 ti gpu. we use the 110m parameter cased bert-base model available in tensorflow hub 3 , with d @xmath0. to align with @xcite , d mark is set to 10, and lstm and mlp hidden state sizes are set to 768 and 300 respectively. dropout rates of 0.1 are applied to bert outputs as well as after relu transforms in mlps. recurrent dropout @xcite with a rate of 0.1 is applied in lstms on hidden states and outputs. to initialize verbnet class embeddings, we use 100-dimensional glove embeddings @xcite averaged over member verbs (d @xmath1. λ vncls is set to 0.5 after a preliminary search over {0.1, 0.5, 1.0}. we follow the fine-tuning methodology described in @xcite , using adam (kingma and ba, 2014) with a batch size of 16. the learning rate is warmed up linearly from 0 to 5e-5 for 10% of training, then decayed linearly to 0 for the rest of training. models are trained for up to 8 epochs. the best-performing checkpoint on the development set, evaluated at every half epoch, is selected for evaluation. unless otherwise mentioned, we train and evaluate all models with at least 7 independent random initializations, and present mean scores in our comparisons. to establish statistical significance, we apply a test for almost stochastic dominance @xcite between test score distributions, using α = 0.05. numbers in bold indicate highest average performance within a given evaluative setting, with a single star indicating statistical significance of almost stochastic dominance over our baseline 3 https://tfhub.dev/google/bert_cased_ l-12_h-768_a-12/1 conll-2005 conll-2012 system wsj brown test peters et al. (2018) 84.6 he et al. (2018) 87.4 80.4 85.5 ouchi et al. (2018) 87.6 78.7 86.2 li et al. (2019) 87.7 80.5 86.0 shi and lin (2019) 88.1 80.9 86.2 our baseline 87.5±0.2 81.2±0.4 86.2±0.1 ## analysis monosemous vs. polysemous predicates to understand the impact of verbnet class features, we break down our evaluation by polysemous and monosemous verbs in why does all classes improve performance on out-of-domain data for monosemous verbs, but not polysemous verbs? intuitively, the per-verb distributions of verbnet classes may change considerably between two domains. using a correctlypredicted class may help mitigate errors on verbs for which one class was dominant during training, but a different class or set of classes are observed during testing in the new domain. this benefit would not be observed with all classes as for a given verb, the same classes used as model inputs during training would be used as inputs on out-ofdomain data. however, vn classes receive fewer updates during training when using only predicted classes. thus, verbs appearing in classes that never or rarely appeared during training will not benefit from predicted class features. all classes may mitigate this issue, since even if a specific class does not appear in the training data, it still can receive updates from examples of polysemous member verbs that belong to other classes (and improved performance over predicted class on monosemous verbs on the out-of-domain brown test set supports this). as future work, a promising direction may therefore be to combine predicted class and all classes features. out-of-vocabulary predicates how well do models incorporating verbnet features generalize on out-of-vocabulary and rare predicates? we split an evaluation on the wsj development set into 5 bins by training set predicate frequency (shown in figure 1 ). comparing development f 1 scores for all classes and predicted class models against our baseline model, we note that vn classes improve srl performance most for predicates appearing 0-50 times in the training data, which account for 24.4% of instances in the development set. focusing on low-frequency predicates, we further divide our evaluation of predicates occurring fewer than 50 times in the training data into 6 bins, one of which is reserved for oov predicates (figure 2 ). from this analysis, we find that vn classes are most impactful on predicates appearing fewer than 10 times in the training data, with a large improvement over the baseline on oov predicates when applying predicted classes. ## conclusions and future work we investigate verbnet classes as an effective level of abstraction for predicates when performing semantic role labeling. we find that incorporating features based on gold verbnet classes improves both verbnet and propbank srl, but when predicted classes are used, this effect is only observed for verbnet. an improvement is also observed without explicit prediction of classes by including a list of all verbnet classes the target predicate belongs to as features. breaking down our evaluation into polysemous and monosemous predicates, we find that predicted classes help more on outof-domain polysemous predicates, while using all 0-49 50-99 100-499 500-899 900+ valid verbnet classes helps more on out-of-domain low-frequency predicates. in multi-task learning experiments motivated by the interdependence of vn classification and srl, we find that joint training improves both tasks when conditioning role labeling on predicted predicates, facilitating vn semantic parsing. in future work, we will investigate alternative approaches incorporating the structure of verbnet into the parsing of verbnet semantic representations. finally, we hope to expand our evaluations to larger, more diverse datasets to further investigate domain transfer.