High-order Semantic Role Labeling (2020)

Abstract: Semantic role labeling is primarily used to identify predicates, arguments, and their semantic relationships. Due to the limitations of modeling methods and the conditions of pre-identified predicates, previous work has focused on the relationships between predicates and arguments and the correlations between arguments at most, while the correlations between predicates have been neglected for a long time. High-order features and structure learning were very common in modeling such correlations before the neural network era. In this paper, we introduce a high-order graph structure for the neural semantic role labeling model, which enables the model to explicitly consider not only the isolated predicate-argument pairs but also the interaction between the predicate-argument pairs. Experimental results on 7 languages of the CoNLL-2009 benchmark show that the high-order structural learning techniques are beneficial to the strong performing SRL models and further boost our baseline to achieve new state-of-the-art results.

## introduction linguistic parsing seeks the syntactic/semantic relationships between language units, such as words or spans @xcite . the algorithms usually use factored representations of graphs to accomplish the target: a set of nodes and relational * corresponding author. this paper was partially supported by national key research and development program of china (no. 2017yfb0304100), key projects of national natural science foundation of china (u1836222 and 61733011), huawei-sjtu long term ai project, cuttingedge machine reading comprehension and language model. rui wang was partially supported by jsps grant-in-aid for early-career scientists (19k20354): "unsupervised neural machine translation in universal scenarios" and nict tenuretrack researcher startup fund "toward intelligent machine translation". before the introduction of deep neural networks, in syntactic parsing (a kind of linguistic parsing), several works @xcite @xcite @xcite showed that high-order parsers utilizing richer factorization information achieve higher accuracy than low-order ones due to the extensive decision history that can lead to significant improvements in inference @xcite . semantic role labeling (srl) @xcite @xcite captures the predicate-argument structure of a given sentence, and it is defined as a shallow semantic parsing task, which is also a typical linguistic parsing task. recent high-performing srl models @xcite @xcite @xcite , whether labeling arguments for a single predicate using sequence tagging model at a time or classifying the candidate predicateargument pairs, are (mainly) belong to first-order parsers. high-order information is an overlooked potential performance enhancer; however, it does suffer from an enormous spatial complexity and an expensive time cost in the inference stage. as a result, most of the previous algorithms for highorder syntactic dependency tree parsing are not directly applicable to neural parsing. in addition, the target of model optimization, the high-order relationship, is very sparse. it is not as convenient for training the model with negative likelihood as the first-order structure is because the efficient gradient backpropagation of parsing errors from the high-order parsing target is indispensable in neural parsing models. bell , in los angeles , and electronic , computer and based 01 makes 01 distributes 01 building 01 a1 am-loc a0 a1 products 01 a0 a1 a2 a0 a1 a3 cop sib gp p a 1 a 2 sibling (sib) a p 2 p 1 co-parent (cop) a p 2 p 1 grandparent (gp) figure 1 : left is the second-order parts (structures) considered in this paper, where the p stands for a predicate, and a stands for argument. right is an example of semantic role labeling from the conll-09 training dataset. we conduct experiments on popular english and multilingual benchmarks. from the evaluation results on both test and out-of-domain sets, we observe a statistically significant increase in semantic-f 1 score with the second-order enhancement and report new state-of-the-art performance in all test set of 7 languages except for the out-of-domain test set in english. additionally, we also evaluated the results of the setting without pre-identified predicates and compared the effects of every different high-order structure combination on all languages to explore how the high-order structure contributes and how its effect differs from language to language. our analysis of the experimental results shows that the explicit higher-order structure learning yields steady improvements over our replicated strong bert baseline for all scenarios. ## high-order structures in srl high-order features or structure learning is known to improve linguistic parser accuracy. in dependency parsing, high-order dependency features encode more complex sub-parts of a dependency tree structure than the features based on first-order, bigram head-modifier relationships. the clear trend in dependency parsing has shown that the addition of such high-order features improves parse accuracy @xcite @xcite @xcite . we find that this addition can also benefit semantic parsing, as a tree is a specific form of a graph, and the high-order properties that exist in a tree apply to the graphs in semantic parsing tasks as well. for a long time, srl has been formulated as a sequential tagging problem or a candidate pair (word pair) classification problem. in the pattern of sequential tagging, only the arguments of one single predicate are labeled at a time, and a crf layer is generally considered to model the relationship between the arguments implicitly @xcite . in the candidate pair classification pattern, @xcite propose an end-to-end approach for jointly predicting all predicates, arguments, and their relationships. this pattern focuses on the first-order relationship between predicates and arguments and adopts dynamic programming decoding to enforce the arguments' constraints. from the perspective of existing srl models, high-order information has long been ignored. although current first-order neural parsers could encode the highorder relationships implicitly under the stacking of the self-attention layers, the advantages of explicit modeling over implicit modeling lie in the lower training cost and better stability. this performance improvement finding resultant of high-order features or structure learning suggests that the same benefits might be observed in srl. thus, this paper intends to explore the integration and effect of high-order structures learning in the neural srl model. the trade-offs between rich high-order structures (features), decoding time complexity, and memory requirements need to be well considered, especially in the current neural models. the work of @xcite suggests that with the help of deep neural network design and training, exact decoding can be replaced with an approximate decoding algorithm, which can significantly reduce the decoding time complexity at a very small performance loss; however, using high-order structure unavoidably brings problematically high graphic memory demand due to the gradient-based learning methods in the neural network model. given an input sentence with length l, order j of parsing model, the memory required is o(l j+1 ). in the current gpu memory conditions, second-order @xmath0. therefore, we enumerate all three second-order structures as objects of study in srl, as shown in the left part of figure 1 , namely sibling (sib), co-parents (cop), and grandparent (gp). as shown in the srl example presented in right part of figure 1 , our second-order srl model looks at several pairs of arcs: though some high-order structures have been studied by some related works @xcite @xcite @xcite in japanese predicate argument structure (pas) @xcite analysis and english srl @xcite , the integration of multiple high-order structures into a single framework and exploring the high-order effects on multiple languages, different high-order structure combinations in a comprehensive way on popular conll-2009 benchmark is the first considered in this paper and thus takes the shape of the main novelties of our work. ## related work the conll-2009 shared task advocated performing srl for multiple languages to promote multilingual nlp applications. @xcite proposed an integrated approach by exploiting large-scale feature sets, while @xcite used a generic feature selection procedure, which yielded significant gains in the multilingual srl shared task. with the development of deep neural networks @xcite @xcite @xcite @xcite for nlp, most subsequent srl works have focused on improving the performance of english, with occasional comparisons to other languages @xcite @xcite @xcite @xcite . @xcite built a polyglot semantic role labeling system by combining resources from all languages in the conll-2009 shared task for exploiting the similarities between semantic structures across languages. this approach, while convenient, is still far less effective than separate model training on different languages. @xcite modeled interactions between argument labeling decisions with a structure refinement network, resulting in an effective model, and outperforming strong factorized baseline models on all 7 languages. @xcite boosted multilingual srl performance with special focus on the impact of syntax and contextualized word representations and achieved new state-ofthe-art results on the conll-2009 benchmarks of all languages, resulting in an effective model and outperforming strong factorized baseline models on all 7 languages high-order parsing is one of the research hotspots in which first-order parsers meet performance bottlenecks; this has been extensively studied in the literature of syntactic dependency parsing @xcite @xcite @xcite @xcite . in semantic parsing, @xcite proposed a way to encode high-order parts with hand-crafted features and introduced a novel co-parent part for semantic dependency parsing. @xcite proposed a quasi-second-order semantic dependency parser with dynamic programming. @xcite trained a second-order parser in an end-toend manner with the help of mean field variational inference and loopy belief propagation approximation. in srl or related research field, there is also some related work on the improvement of performance by high-order structural information. on the japanese naist predict-argument structure (pas) dataset, some works @xcite @xcite @xcite mainly studied the relationship between multiple predicates separately, that is, the gp and cp high-order relationship mentioned in our paper. @xcite considered the interactions between predicate-argument pairs on chinese propbank dataset. although the motivation is consistent with our work, we first consider multiple high-order relationships at the same time within a more uniform framework on more popular benchmarks and for more languages. ## conclusion and future work in this work, we propose high-order structure learning for dependency semantic role labeling. the proposed framework explicitly models high-order graph structures on a strong first-order baseline model while scoring the correlation of predicted predicate-argument pairs. the resulting model achieves state-of-the-art results on all 7 languages in the conll-2009 test data sets except the outof-domain benchmark in english. in addition, we consider both given and not-given predicates on all languages, explore the impact of every high-order structure combinations on performance for all languages, and reveal the adaptive range of high-order structure learning on different languages. in future work, we will continue to explore higher-order structures and pruning strategies to reduce the time complexity and memory occupation.