Modeling Syntactic-Semantic Dependency Correlations in Semantic Role Labeling Using Mixture Models (2022)

Abstract: In this paper, we propose a mixture model-based end-to-end method to model the syntactic-semantic dependency correlation in Semantic Role Labeling (SRL). Semantic dependencies in SRL are modeled as a distribution over semantic dependency labels conditioned on a predicate and an argument word. The semantic label distribution varies depending on Shortest Syntactic Dependency Path (SSDP) hop patterns. We target the variation of semantic label distributions using a mixture model, separately estimating semantic label distributions for different hop patterns and probabilistically clustering hop patterns with similar semantic label distributions. Experiments show that the proposed method successfully learns a cluster assignment reflecting the variation of semantic label distributions. Modeling the variation improves performance in predicting short distance semantic dependencies, in addition to the improvement on long distance semantic dependencies that previous syntax-aware methods have achieved. The proposed method achieves a small but statistically significant improvement over baseline methods in English, German, and Spanish and obtains competitive performance with state-of-the-art methods in English.

## introduction semantic role labeling (srl) answers an essential question about sentence semantics: " @xcite [does what] [to whom]". a core problem of srl is identifying semantic dependencies that specify the semantic role of arguments in relation to predicates @xcite . for example, [who] (argument) is the agent (semantic role) to [does what] (predicate). semantic dependency parsers @xcite identify semantic dependencies by giving a distribution over semantic dependency labels (denoted as semantic label distribution) for all predicate-argument pairs. 1 our code is available at this repository. in this paper, we propose a mixture model (pearson, 1894) based semantic dependency parser for srl where we target the dependence of semantic label distributions on shortest syntactic dependency path (ssdp) patterns. ssdp is the shortest path connecting a predicate-argument pair in a syntactic dependency tree. @xcite and @xcite claim that ssdp encodes most information about bigram relations, such as the semantic dependency. indeed, previous research @xcite shows that modeling the correlation between ssdps and semantic dependencies is crucial for building a high-performance srl system. semantic label distributions vary depending on ssdps, even when the ssdps connect predicateargument pairs with the same surface words. figure 1 shows an example where two predicate-argument pairs have different semantic dependency labels while sharing the same surface words. ssdp patterns help discriminate semantic labels between the two pairs. the example indicates the dependence of semantic label distributions on ssdp patterns. we propose a mixture model-based method (figure 2 ) to model the dependence in two steps: (1) separately estimating semantic label distributions for different ssdp patterns as component distributions, and (2) probabilistically clustering ssdp patterns with similar semantic label distributions using a mixture weight. the mixture model estimates the semantic label distribution by aggregating the component distributions using the mixture weight. we focus on ssdp hop patterns in this paper as we observed a drastic variation in semantic label distributions for different hop patterns through the change in mutual information @xcite (section 2). we evaluate the proposed method using the conll-2009 dataset @xcite foot_0 , the most popular multi-lingual srl dataset with parallel syntactic and semantic dependency annotations. experiments show that the proposed method correctly learns a mixture weight reflecting the variation in semantic label distributions. modeling the variation improves performance in predicting short distance semantic dependencies in addition to long distance dependencies that previous syntax-aware methods @xcite @xcite improve only on. previous syntax-aware methods improve their performance on long distance dependencies at the expense of the performance on short distance dependencies. in comparison, the proposed method makes no such compromise, improving its performance over semantic dependencies of all ranges. in general, the proposed method obtains a small but statistically significant improvement over baseline methods in english, german, and spanish and achieves competitive performance with state-of-the-art methods in english. our contributions are: (1) studying the variation in semantic label distributions for different ssdp hop patterns, (2) proposing a mixture model-based method capturing the variation, and (3) conducting a detailed experiment evaluating the proposed method. ## motivation as mentioned in section 1, ssdp affects the choice of semantic dependency labels. we study the impact of ssdp hop patterns on semantic label distributions through the change in mutual information @xcite in this section. we observe a drastic change in mutual information only for hop patterns that frequently co-occur with semantic dependencies. ssdp is the path connecting a predicateargument pair in a syntactic dependency tree. its hop pattern describes the number of transitions needed to transit from the predicate to the argument. we denote the hop pattern by (α, β), where α is the number of dependent-to-head transitions and β is the number of head-to-dependent transitions. in a syntactic dependency tree, syntactic dependencies are arcs pointing from syntactic heads to syntactic dependents. the head-to-dependent transition moves in the same direction as the syntactic dependencies, whereas the dependent-to-head transition moves in the opposite direction. in figure 1 , the ssdp connecting "eliminate" and "it" consists of a dependent-to-head transition moving from "eliminate" to "will", and a head-to-dependent transition moving from "will" to "it". the hop pattern of this ssdp is (1, 1). we denote the syntactic random variable for hop patterns as x and the semantic random variable for semantic labels as y . x maps predicate-argument word pairs (p s , a s ) in a sentence s to their hop patterns, whereas y maps the pairs to their semantic labels. their mutual information mi(x, y ) measures the reduction in uncertainty about y after knowing x. high mutual information indicates relatively low uncertainty in the conditional distribution p y |x . to highlight the impact of hop patterns on semantic label distributions, we compare the mutual information of two ideal models, a syntax-aware model (x (α,β) , y ) and a syntax-agnostic model (x 0 , y ). we define the syntactic variables x (α,β) and x 0 as equation 1 and 2. this definition makes the variable x (α,β) sensitive only to the hop pattern (α, β) and x 0 blind to any hop pattern information. we define the mutual information gain of (α, β) as the difference in mutual information between the syntax-aware model and the syntax-agnostic model (equation 3 ). figure 3 reports the mutual information gain of each hop pattern using the english training set of the conll-2009 dataset. the figure shows that different hop patterns have drastically varying mutual information gains. a sharp spike of mutual information gain occurs in the hop pattern (0, 1) with a gain value of 0.149 bits, indicating a strong impact of the hop pattern (0, 1) on semantic label distributions. hop patterns with relatively short transitions have non-zero gains ranging from 0.011 bits to 0.149 bits, which indicates the degree of impact differs drastically. these hop patterns frequently co-occur with semantic dependencies @xcite . on the other hand, hop patterns cooccurring rarely with semantic dependencies have long transitions. these hop patterns have near-zero mutual information gains in figure 3 , which indicates the weak impact of the patterns. the varying degree of impact motivates the separate estimation of semantic label distributions for different hop patterns. the amount of hop patterns with a weak impact motivates the clustering of hop patterns that share similar semantic label distributions. ## background in this section, we present background information about syntactic and semantic dependency parsing and mixture models. we also present a brief survey about syntax-aware srl methods using ssdp information and compare the proposed method with the previous methods. ## proposal in this section, we present the proposed mixture model-based semantic dependency parser to model the dependence of semantic label distributions on ssdp hop patterns. in section 2, we discussed the need to separately estimate semantic label distributions for different hop patterns and the need to cluster hop patterns sharing similar semantic label distributions. the proposed parser estimates semantic label distributions for different hop patterns using the component distributions and clusters hop patterns using the mixture weight of a mixture model. figure 2 illustrates the model architecture of the proposed method. the model contains a conventional biaffine parser for syntactic dependencies and a mixture model-based biaffine parser for semantic dependencies. the syntactic parser provides a syntactic dependency tree from which the clustering component extracts hop patterns and determines the mixture weights. the biaffine parsers in the semantic parser estimate the component distributions. the semantic parser computes the semantic label distribution by aggregating the component distributions using the mixture weight. the syntactic and the semantic parser share a backbone sentence encoder, a transformer model in our implementation. we jointly optimize the parameters of the syntactic and the semantic parser by optimizing the log-likelihood of the syntactic dependencies and a variational lowerbound of the log-likelihood (elbo) of the semantic dependencies. we use the lowerbound as an approximation to the loglikelihood for inference because we find it works best in predicting semantic dependencies. we expand on the training objective of the semantic parser. the objective is to maximize the likelihood p θ (r|p, a) of the observed semantic label r conditioned on the predicate p and the argument a. we rewrite the likelihood as a marginal of the joint likelihood p θ (r, c|p, a) where c is the index of the component distributions. the joint likelihood can be decomposed as equation 12 where the former term corresponds to the component distributions and the latter term corresponds to the mixture weight. since we are interested in separating semantic label distributions by hop patterns, we replace the term p θ (c|p, a) with p θ (c|ssdp(p, a)) where ssdp(p, a) maps predicate-argument pairs to their hop patterns. p θ (c|ssdp(p, a)) also serves as the variational approximation q(c|r, p, a) because we assume the hop pattern, together with the predicate-argument pair, determines the semantic dependency label. this assumption removes the need to condition the component index c on the semantic label r in the variational approximation q. in this implementation, we encode hop patterns with orthogonally initialized embeddings and estimate the mixture weight of a hop pattern by applying a multi-layer perceptron followed by a softmax layer to the embedding. equation 16 depicts the full objective of the pro-posed model. it consists of a log-likelihood objective of the syntactic parser (equation 17 ) and the elbo objective of the semantic parser (equation 15 ). g syn stands for the set of all syntactic dependencies (h, d), whereas g sem stands for the set of all semantic dependencies (r, p, a). ## conclusion this paper presented a mixture model-based method for syntax-aware semantic dependency parsing in srl. the method models the dependence of semantic label distributions on ssdp patterns. we focused on ssdp hop patterns because we observed a drastic variation in semantic label distributions through the change in mutual information. the proposed method successfully learned a mixture weight reflecting the variation. the method improved performance in predicting both short and long distance semantic dependencies, whereas baseline syntax-aware methods improved only on long distance dependencies. the method outperformed baseline methods by a small