Transformer- GCRF : Recovering C hinese Dropped Pronouns with General Conditional Random Fields (2020)

Abstract: Pronouns are often dropped in Chinese conversations and recovering the dropped pronouns is important for NLP applications such as Machine Translation. Existing approaches usually formulate this as a sequence labeling task of predicting whether there is a dropped pronoun before each token and its type. Each utterance is considered to be a sequence and labeled independently. Although these approaches have shown promise, labeling each utterance independently ignores the dependencies between pronouns in neighboring utterances. Modeling these dependencies is critical to improving the performance of dropped pronoun recovery. In this paper, we present a novel framework that combines the strength of Transformer network with General Conditional Random Fields (GCRF) to model the dependencies between pronouns in neighboring utterances. Results on three Chinese conversation datasets show that the Transformer-GCRF model outperforms the state-of-the-art dropped pronoun recovery models. Exploratory analysis also demonstrates that the GCRF did help to capture the dependencies between pronouns in neighboring utterances, thus contributes to the performance improvements.

## introduction in pro-drop languages such as chinese, pronouns can be dropped as the identity of the pronoun can be inferred from the context, and this happens more frequently in conversations @xcite . recovering dropped pronouns (dps) is a critical task for many nlp applications such as machine translation where the dropped pronouns need to be translated explicitly in the target language @xcite . recovering dropped pronoun is different from traditional pronoun resolution tasks @xcite : a conversation snippet between participant a and b. the dropped pronouns are shown in the brackets, and the dialogue patterns are marked with blue arrows. 2018), which aim to resolve the anaphoric pronouns to their antecedents. in dropped pronoun recovery, we consider both anaphoric and nonanaphoric pronouns, and we do not directly resolve the dropped pronoun to its antecedent, which is infeasible for non-anaphoric pronouns. we recover the dropped pronoun as one of 17 types pronouns pre-defined in @xcite , which include five types of abstract pronouns corresponding to the non-anaphoric pronouns. thus traditional rulebased pronoun resolution methods are not suitable for recovering dropped pronouns. existing approaches formulate dropped pronoun recovery as a sequence labeling task of predicting whether a pronoun has been dropped before each token and the type of the dropped pronoun. for example, @xcite first studied this problem in sms data and utilized a maximum entropy classifier to recover dropped pronouns. deep neural networks such as multi-layer perceptrons (mlps) and structured attention networks have also been used to tackle this problem @xcite . @xcite used a linear-chain crf to model the dependency be-tween the sequence of predictions in a utterance. although these models have achieved various degrees of success, they all assume that each utterance in a conversation should be labeled independently. this practice overlooks the dependencies between dropped pronouns in neighboring utterances, and results in sequences of predicted dropped pronouns are incompatible with one another. we illustrate this problem through an example in figure 1 , in which the dropped pronouns are shown in brackets. the pronoun can be dropped as a subject at the beginning of a utterance, or as an object in the middle of a utterance. pronouns dropped at the beginning of consecutive utterances usually have strong dependencies that pattern with three types of dialogue transitions (i.e., reply, expansion and acknowledgment) presented in @xcite . for example, in figure 1 , the pronoun in the second utterance b 1 is "我 (i)", the dropped pronoun in the third utterance b 2 should also be "我 (i)" since b 2 is an expansion of b 1 by the same speaker. thus modeling the dependency between pronouns in adjacent sentences is helpful to recover pronoun dropped at utterance-initial positions. in contrast, the pronoun "他 (him)" dropped as an object in utterance b 4 should be recovered by capturing referent semantics from the context and modeling token dependencies in the same utterance. to model the dependencies between predictions in the conversation snippet, we propose a novel framework called transformer-gcrf that combines the strength of the transformer model @xcite in representation learning and the capacity of general conditional random fields (gcrf) to model the dependencies between predictions. in the gcrf, a vertical chain is designed to capture the pronoun dependencies between the neighboring utterances, and horizontal chains are used for modeling the prediction dependencies inside each utterance. in this way, transformer-gcrf successfully models the cross-utterance pronoun dependencies as well as the intra-utterance prediction dependencies simultaneously. experimental results on three conversation datasets show that transformer-gcrf significantly outperforms the state-of-the-art recovery models. we also conduct ablative experiments that demonstrate the improvement in performance of our transformer-gcrf model derives both from the transformer encoder and the ability of gcrf layer to model the dependencies between dropped pronouns in neighboring utterances. all code is available at https://github. com/ningningyang/transformer-gcrf . the major contributions of the paper are summarized as follows: the model jointly predicts all dropped pronouns in an entire conversation snippet. ## our approach: transformer-gcrf we start by formalizing the dropped pronoun recovery task as follows. given a chinese conversation snippet @xmath0, • • • , x n ) which consists of n pro-drop utterances, where the i-th utterance x @xmath1, • • • , x im i ) is a sequence of m i tokens, and additionally given a set of k possible labels @xmath2, • • • , y k-1 } ∪ {none} where each y j corresponds to a pre-defined pronoun @xcite or 'none', which means no pronoun is dropped, the goal of our task is to assign a label @xmath3. we model this task as the problem of maximizing the conditional probability p(y|x), where y is the label sequence assigned to the tokens in x. the conditional probability of a label assignment y given the whole conversation snippet x can be written as: ## datasets and experimental setup datasets: we evaluate the performance of transformer-gcrf on three conversation benchmarks: chinese text message dataset (sms), ontonotes release 5.0, and baiduzhidao. the sms dataset is described in @xcite and contains 684 text message documents generated by users via sms or chat. following @xcite , we reserved 16.7% of the training set as the development set, and a separate test set was used to evaluate the models. the ontonotes release 5.0 was released in the conll 2012 shared task. we used the tc section which consists of transcripts of chinese telephone conversation speech. the baiduzhidao dataset is a question answering dialogue corpus collected by @xcite . ten types of dropped pronouns are annotated according to the pronoun annotation guidelines. the statistics of these three benchmarks are reported in baselines: state-of-the-art dropped pronoun recovery models are used as baselines: (1) mepr @xcite which leverages a set of elaborately designed features and trains a maximum entropy classifier to predict the type of dropped pronoun before each token; (2) nrm @xcite which employs two separate mlps to predict the position and type of a dropped pronoun utilizing representation of words in a fixed-length window; (3) bigru which utilizes a bidirectional rnn to encode each token in a pro-drop sentence and makes prediction based on the encoded states; (4) ndpr @xcite which models dropped pronoun referents by attending to the context and independently predicts the presence and type of dp for each token. we also compare three variants of transformer-gcrf as: (1) transformer-gcrf(w/o refine) which removes step 2 in section 3.3.1 during the graph construction process, for exploring the effectiveness of processing ovp and interjections; (2) transformer which removes the whole gcrf layer that globally optimizes the prediction sequences, and directly adds a mlp layer on the top of transformer encoder to predict the dropped pronouns. it aims to explore the contribution of transformer encoder among the total effectiveness of transformer- ## conclusion and future work in this paper, we presented a novel model for recovering the dropped pronouns in chinese conversations. the model, referred to as transformer-gcrf, formulates dropped pronoun recovery as a sequence labeling problem. transformer is employed to represent the utterances and gcrf is used to make the final predictions, through capturing both cross-utterance and intra-utterance dependencies between pronouns. experimental results on three chinese conversational datasets show that transformer-gcrf consistently outperforms stateof-the-art baselines. in the future, we will do some extrinsic evaluation by applying our proposed model in some downstream applications like pronoun resolution, to further explore the effectiveness of modeling cross-utterance dependencies in practical applications.