Semantic-aware C hinese Zero Pronoun Resolution with Pre-trained Semantic Dependency Parser (2020)

Abstract: Deep learning-based Chinese zero pronoun resolution model has achieved better performance than traditional machine learning-based model. However, the existing work related to Chinese zero pronoun resolution has not yet well integrated linguistic information into the deep learningbased Chinese zero pronoun resolution model. This paper adopts the idea based on the pre-trained model, and integrates the semantic representations in the pre-trained Chinese semantic dependency graph parser into the Chinese zero pronoun resolution model. The experimental results on OntoNotes-5.0 dataset show that our proposed Chinese zero pronoun resolution model with pretrained Chinese semantic dependency parser improves the F-score by 0.4% compared with our baseline model, and obtains better results than other deep learning-based Chinese zero pronoun resolution models. In addition, we integrate the BERT representations into our model so that the performance of our model was improved by 0.7% compared with our baseline model.

## introduction chinese zero pronoun resolution is a special task of coreference resolution @xcite . its purpose is to find the real referent of the omitted parts with syntactic functions in the text. these omitted parts are usually called zero pronouns, and their real referents are called antecedents. below is a sentence with zero pronouns: in this example, the referent of zero pronoun * pro 1 * is /•/i0, and the referent of zero pronoun * pro 2 * is /¨/her0. since the zero pronoun is not a real word in the text, its resolution is much more difficult than that of the overt pronoun. the existence of zero pronouns poses challenges for machines to automatically understand text. the existing chinese zero pronoun resolution models with better performance usually adopt the method of deep learning @xcite ; @xcite ; @xcite ; @xcite ; @xcite . the deep learning-based methods can make the model automatically extract the taskrelated distributed representations through end-to-end training, thereby avoiding the problem that traditional machine learning-based methods rely heavily on artificially designed feature templates @xcite . however, it is difficult for deep learning-based models to encode effective syntactic, semantic and other linguistic information only through end-to-end training. many deep learning-based chinese zero pronoun resolution models still use syntactic features extracted from the syntactic parsing tree as a supplement to distributed representations. intuitively, semantic information as a higher level linguistic information is also very important to the chinese zero pronoun resolution task, however few studies have attempted to integrate semantic information into the chinese zero pronoun resolution model. therefore, how to effectively integrate semantic information into the chinese zero pronoun resolution model is a challenging problem. with the development of semantic parsing, the performance of some sentence-level semantic parsers have made remarkable progress, which provides opportunities for the application of sentence-level semantic parsing in other natural language processing tasks. in this paper, we proposed a semantic-aware chinese zero pronoun resolution model that integrates the semantic information from pre-trained chinese semantic dependency graph parser. chinese semantic dependency graph parsing @xcite is a semantic-level dependency parsing task, which is an extension of syntactic dependency parsing. each node in the semantic dependency graph represents a word in the sentence, and the nodes are connected by directed edges with semantic relationship labels. figure 1 is an example of a chinese semantic dependency graph. the realization of our model requires two stages. in the first stage, we use the chinese semantic dependency graph parsing as a pre-training task to obtain a pre-trained semantic dependency graph parser. in the second stage, we feed the sentence which will be processed into the pre-trained semantic dependency graph parser to obtain the semantic-aware representations, and integrate these implicit semantic information into the chinese zero pronoun resolution model. we implement a attention-based chinese zero pronoun resolution model as our baseline model. the experiments on ontonotes-5.0 dataset show that our proposed chinese zero pronoun resolution model with pre-trained chinese semantic dependency parser improves the f-score by 0.4% compared with our baseline model, and obtains better results than other deep learning-based chinese zero pronoun resolution models. in addition, we integrate the bert representations into our model so that the performance of our model was improved by 0.7% compared with our baseline model. ## method given the success of the method of pre-training of syntactic dependency parser in some natural language processing tasks, we adopt a similar method to take the chinese semantic graph dependency parsing as a pre-training task, and apply this method to chinese zero pronoun resolution task. our proposed chinese zero pronoun resolution model with pre-trained chinese semantic dependency parser is composed of two parts, one is the pre-trained chinese semantic dependency graph parser and the other is the chinese zero pronoun resolution model. specifically, the chinese semantic dependency graph parser consists of two parts: bilstm-based encoder and biaffine-based decoder. the chinese zero pronoun resolution model consists of three parts: the zero pronoun module(zp module), the candidate antecedents module (ca module) and the discrimination module. in addition, in order to obtain sentence-level semantic representations, we also used a cnn-based sentence representation extractor(sr extractor). for a sentence to be processed, the representations of each word will be feed into the pre-trained chinese semantic dependency graph parser, so that each word can obtain the semantic-aware representations containing the information of semantic dependency graph. then, the semantic-aware representations will be integrated into the chinese zero pronoun resolution model to perform the subsequent processing. the overall architecture of our proposed model is shown in figure 2 : computational linguistics tasks related to semantic dependency graph parsing. @xcite ; @xcite in the process of pre-training, we first use the concatenation of word vector, part of speech vector and character-level vector to represent a word. then, we feed the word representations into the encoder to obtain the context-aware representations. finally, we feed the context-aware representations of the word into the decoder to calculate the score of the dependency arc in the semantic dependency graph. the complete calculation process of the semantic dependency graph parser is shown in the following formulas: