Coupling Context Modeling with Zero Pronoun Recovering for Document-Level Natural Language Generation (2021)

Abstract: Natural language generation (NLG) tasks on pro-drop languages are known to suffer from zero pronoun (ZP) problems, and the problems remain challenging due to the scarcity of ZP-annotated NLG corpora. In this case, we propose a highly adaptive two-stage approach to couple context modeling with ZP recovering to mitigate the ZP problem in NLG tasks. Notably, we frame the recovery process in a task-supervised fashion where the ZP representation recovering capability is learned during the NLG task learning process, thus our method does not require NLG corpora annotated with ZPs. For system enhancement, we learn an adversarial bot to adjust our model outputs to alleviate the error propagation caused by mis-recovered ZPs. Experiments on three document-level NLG tasks, i.e., machine translation, question answering, and summarization, show that our approach can improve the performance to a great extent, and the improvement on pronoun translation is very impressive.

## introduction for a long time, natural language generation (nlg) has attracted a lot of attention for its importance in serving human life. in the literature, although various studies have been done to bridge the discrepancy between human and machine, document-level nlg (d-nlg) tasks still suffer from cohesion issues caused by zero pronoun (zp). as a discourse phenomenon where pronouns can be omitted when they are pragmatically or grammatically inferable from context @xcite , zero pronoun appears frequently in pro-drop languages like chinese, spanish, etc. taking the chinese ted corpus as an example, according to our statistics, in sentences with an average length of 18, each sentence will omit around 0.5 pronouns. facing this problem, lots of attention has been paid to zp resolution in the past decade @xcite @xcite @xcite , and these studies have achieved certain success. nevertheless, due to the lack of nlg corpora annotated with zps, the problem of zero pronoun remains challenging in document-level nlg tasks. recently, more and more researchers turn to zp resolution in nlg tasks @xcite @xcite . among these studies, one line explicitly deals with the zp problem by recovering dropped pronouns through either annotated corpora @xcite @xcite or pre-trained zp resolution systems @xcite ). another line indirectly deals with the zp problem by producing better discourse cohesion through document context modeling @xcite @xcite . although the above studies have made great progresses, zp resolution in nlg still faces the following possible bottlenecks: (i) zp-annotated corpora tailored for nlg tasks are scarcity, and existing zp corpora are limited to certain domains and tasks. (ii) using pre-trained zp resolution systems to recover pronoun labels for nlg tasks could lead to the notorious error propagation problem. (iii) although document context modeling can improve discourse-level cohesion to some extent, the context information is too broad to solve the zp problem in a targeted manner. in order to solve the zp problem in d-nlg tasks while avoiding the above disadvantages, we introduce a highly adaptive two-stage approach that couples context modeling with zp recovering. specifically, our approach mainly consists of two phases: first, we pre-train a fault-tolerant zp position detector for downstream tasks' data corpora to automatically detect zp positions. second, we perform document context modeling for both tasksupervised zp recovering and zp-focused nlg task learning. notably, the zp recovering process and the nlg task learning process depend on and promote with each other harmoniously. on the one hand, instead of recovering specific pronoun labels, we learn zp representation through the supervision of nlg tasks and thus our method does not require large-scale zp-annotated data. one the other, the achieved zp representation is further fused into the previously modeled document context for zpfocused nlg task learning in turn. in this way, the recovered zp representation and the supervision of specific tasks are well shared between the two processes for high-quality model integration. to comprehensively investigate our proposed method, we conduct experiments foot_0 on three d-nlg tasks: document-level neural machine translation (nmt), question answering (qa), and summarization. experimental results show that our approach can significantly improve the performance on these tasks due to the effective combination of zp recovering and document context modeling. furthermore, we use both apt @xcite and crc @xcite to evaluate our model performance on pronoun generation, and the results show that our approach can achieve remarkable performance. ## related work as a fundamental research in natural language processing, zp resolution aims at detecting pronoun chains and resolving missing pronouns to their antecedents. in the literature, previous work mainly resolved zps in three steps: zero pronoun detection, anaphoricity determination, and coreference linking. on this basis, varied traditional rule-based or machine-learning methods were used for zp resolution @xcite @xcite . recently, some neural approaches @xcite @xcite were proposed and have achieved certain success due to their better objective representation and powerful neural architectures. as a common language phenomenon in prodrop languages, zero pronoun could result in poor discourse-level cohesion and thus seriously impact the performance of document-level nlg. to date, two types of researches have been conducted to alleviate the discourse-level cohesion deficiency: (i) recovering dropped pronouns in specific nlg corpora for downstream tasks; (ii) using well-designed context-aware architectures for document-level cohesion modeling. first, some studies directly used manually @xcite @xcite or automatically @xcite annotated zp corpora for nlg tasks. nevertheless, the manual annotation is usually time consuming and the automatic annotation is limited to specific tasks like machine translation, it remains challenging when facing new corpus domains or tasks. moreover, although some two-stage methods were proposed to use pre-trained zp resolution systems for preprocessing @xcite , these methods are known to face notorious error propagation problems. second, some recent studies explored context-aware architectures for better document cohesion modeling @xcite @xcite @xcite . for instance, @xcite proposed a hierarchical model to capture global context, which can significantly improve pronoun translation in document-level nmt. although the above studies can well capture document-level cohesion to some extent, the context information is too broad to solve zp problems in a targeted manner. the difference between our method and previous ones is two-fold: first, our two-stage method is a combination of the above categories which can mitigate both corpora limitation and error propagation issues. second, compared with previous zp recovery methods, we focus on enhancing nlg with the recovered zp representations rather than with specific pronoun labels. therefore, our approach does not require zp-annotated nlg corpora. ## d-nlg with zp recovery in this section, we introduce the proposed highly adaptive approach which consists of two stages, i.e., detecting zp positions in the first stage (section 3.1) and then coupling context modeling with zp representation recovering in document-level nlg (sections 3.2 and 3.3). ## experimentation in this section, we conduct several experiments on document-level nmt, qa, and summarization to evaluate our proposed approach. ## conclusion in this paper, we introduced a highly adaptive twostage method to mitigate the cohesion problem posed by the zp phenomenon in document-level nlg tasks. to tackle both error propagation and corpus limitation issues, we first pre-trained a faulttolerant zp position detector for automatic zp position prediction, and then performed document context modeling for both task-supervised zp recovering and zp-focused nlg task learning. and we also trained our model in an adversarial fashion to alleviate the language generation confusion caused by mis-recovered zps. experiments on three d-nlg tasks show that our approach can greatly improve the performances, and the performance on pronoun translation is remarkable.