Semantic Role Labeling with Heterogeneous Syntactic Knowledge (2020)

Abstract: Recently, due to the interplay between syntax and semantics, incorporating syntactic knowledge into neural semantic role labeling (SRL) has achieved much attention. Most of the previous syntax-aware SRL works focus on explicitly modeling homogeneous syntactic knowledge over tree outputs. In this work, we propose to encode heterogeneous syntactic knowledge for SRL from both explicit and implicit representations. First, we introduce graph convolutional networks to explicitly encode multiple heterogeneous dependency parse trees. Second, we extract the implicit syntactic representations from syntactic parser trained with heterogeneous treebanks. Finally, we inject the two types of heterogeneous syntax-aware representations into the base SRL model as extra inputs. We conduct experiments on two widely-used benchmark datasets, i.e., Chinese Proposition Bank 1.0 and English CoNLL-2005 dataset. Experimental results show that incorporating heterogeneous syntactic knowledge brings significant improvements over strong baselines. We further conduct detailed analysis to gain insights on the usefulness of heterogeneous (vs. homogeneous) syntactic knowledge and the effectiveness of our proposed approaches for modeling such knowledge.

## introduction semantic role labeling (srl) is a fundamental task in natural language processing (nlp), which aims to find the predicate argument structures (who did what to whom, when and where, etc.) in a sentence (see figure 1 as an example). recent srl works can mostly be divided into two categories, i.e., syntax-aware @xcite @xcite and syntax-agnostic @xcite approaches according to whether incorporating syntactic knowledge or not. most syntax-agnostic works employ deep bilstm or self-attention encoder to encode the contextual information of natural sentences, with various kinds of scorers to predict the probabilities of bio-based semantic roles @xcite or predicate-argument-role tuples @xcite . motivated by the strong interplay between syntax and semantics, researchers explore various approaches to integrate syntactic knowledge into syntax-agnostic models. @xcite propose to use dependency-based embeddings in a neural srl model for dependency-based srl. @xcite introduce k-order pruning algorithm to prune arguments according to dependency trees. however, previous syntax-aware works mainly employ singleton/homogeneous automatic dependency trees, which are generated by a syntactic parser trained on a specific syntactic treebank, like penn treebank (ptb) @xcite . our work follows the syntax-aware approach and enhances srl with heterogeneous syntactic knowledge. we define heterogeneous syntactic treebanks as treebanks that follow different annotation guidelines. all is well known, there exist many published dependency treebanks that follow different annotation guidelines, i.e., english ptb @xcite , universal dependencies (ud) @xcite , penn chinese treebank (pctb) @xcite , chinese dependency treebank (cdt) @xcite 全 年 来 琼 旅游 的 游客 达到 七百九十万 人次 throughout the year to hainan traveling tourists reach 7.9 million number dep clf rcmod dobj conj cpm nsubj root nummod range att adv rad vob att rad sbv root att vob figure 2: an example of automatic heterogeneous trees, where the top dependency tree is generated from parser pctb and the bottom dependency tree is generated from parser cdt . in this work, we propose two types of methods from the perspective of explicit and implicit to take advantage of heterogeneous syntactic knowledge, which we believe are highly complementary. our baseline model follows the architecture of @xcite . afterwards, we inject the heterogeneous syntactic knowledge into the base model using two proposed methods. for the explicit method, we try to encode the heterogeneous automatic dependency trees with the recent popular graph convolutional networks (gcn) @xcite . for the implicit method, which is inspired by the powerful representations from pre-trained language models, like @xcite and bert @xcite , we introduce a method to extract implicit syntactic representations from the dependency parser trained with heterogeneous syntactic treebanks. it is well known that the main reason for the success of pre-trained language model representations is the use of large amounts of natural text. however, it is difficult to obtain and costly to annotate large amounts of syntactic data. therefore, making full use of existing heterogeneous data is the most feasible and natural idea. intuitively, the explicit method models the syntactic structure of a sentence, providing valuable syntactic position information, while the implicit method aims to capture the syntactic representation for each word into a vector. these two methods contain different types of syntactic knowledge, which are thus highly complementary. to verify the effectiveness of injecting heterogeneous syntactic knowledge, we conduct experiments on the widely-used chinese and english srl benchmarks, i.e., chinese proposition bank 1.0 and english conll-2005. our contributions are listed as follows: ## base model following @xcite , we treat srl as a predicate-argument-role tuple identification task in our work. formally, given a sentence @xmath0, w 2 , ..., w n , we denote the candidate predicates as @xmath1, w 2 , ..., w n }, the candidate arguments as @xmath2, ..., w j )|1 ≤ @xmath3, and the semantic roles as r. the goal is to predict a set of predicate-argument-role tuples @xmath4. we basically use the framework of @xcite as our baseline model. in general, the model consists of four modules, i.e., input layer, bilstms encoder layer, predicate and argument representation layer, and mlp scorers layer. the middle component of figure 3 shows the architecture of the baseline model. in the following, we briefly introduce the framework of the baseline model. input layer. the input of the i-th word in s is composed of fixed word embedding and fine-tuned char representation. formally, x @xmath5, where ⊕ is the concatenate operation. the char representation rep char w i is generated by cnns on the characters of the i-th word. bilstms encoder layer. the baseline model employs three layer bilstms as the encoder layer, which is enhanced by the highway connections. we denote the i-th output of bilstms as h i . predicate and argument representation layer. the model directly treats the output hidden states from the top bilstm layer as the representations of candidate predicates. the representation of the k-th word as the candidate predicate is denoted as r p @xmath6. the representation of candidate argument is composed of four parts: 1) the bilstm output of the beginning word in the argument, 2) the bilstm output of the end word in the argument, 3) an embedding indicating the length of the argument, and 4) a softmax weighted summation of the bilstm hidden outputs in the range of candidate argument, where the softmax weights are computed by attention mechanism over words in the argument span. formally, for a candidate argument from i-th word to j-th word, its representation is defined as r a i,@xmath7,j , where e i,j is the softmax weighted summation. we use p and a as the abbreviation of predicate and argument. mlp scorers layer. three mlp scorers are employed to compute the scores of the candidate predicates, arguments, and the semantic roles between the predicted predicates and arguments, respectively. the objective is to find the highest-scoring semantic structure, which is computed as: ## syntactic knowledge in this section, we introduce the proposed methods for extracting syntactic knowledge from heterogeneous dependency treebanks. first, we introduce the method for encoding singleton dependency trees and then detailedly describe the variations to extract heterogeneous syntactic knowledge. ## related work recently, srl has achieved significant improvements because of the development of deep learning. previous works can mostly be divided into two kinds of methods, syntax-agnostic methods, which focus on the srl problem itself, and syntax-aware methods, which explore various ways to integrate syntactic knowledge into the srl models. @xcite propose to use deep bilstms for english spanbased srl. he et al. (2017) ## conclusion we propose to encode heterogeneous syntactic knowledge with explicit and implicit methods to help srl. for the explicit aspect, we propose exphdp to encode the heterogeneous automatic dependency trees, which can provide more information compared with singleton automatic dependency trees. for the implicit aspect, we extract implicit syntactic representations from the dependency parser trained with heterogeneous dependency treebanks. experimental results and detailed analyses demonstrate that our proposed methods effectively capture heterogeneous syntactic knowledge, and thus achieve more improvements compared with using singleton dependency trees. we also discuss the limitations of syntactic knowledge, for which we will explore ways to integrate heterogeneous syntactic knowledge into pre-trained language models in the future.