PRINCE : Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training (2022)

Abstract: Pre-trained Language Models (PLMs) have shown effectiveness in various Natural Language Processing (NLP) tasks. Denoising autoencoder is one of the most successful pre-training frameworks, learning to recompose the original text given a noise-corrupted one. The existing studies mainly focus on injecting noises into the input. This paper introduces a simple yet effective pre-training paradigm, equipped with a knowledge-enhanced decoder that predicts the next entity token with noises in the prefix, explicitly strengthening the representation learning of entities that span over multiple input tokens. Specifically, when predicting the next token within an entity, we feed masks into the prefix in place of some of the previous ground-truth tokens that constitute the entity. Our model achieves new state-of-the-art results on two knowledge-driven data-to-text generation tasks with up to 2% BLEU gains.

## introduction pre-trained language models (plms), such as bert @xcite , mass @xcite , bart @xcite , and t5 @xcite , have had remarkable performances in various natural language processing (nlp) tasks thanks to the use of denoising autoencoder pretraining schema that is optimized to reconstruct the original text given a noise-corrupted one. despite its common usage, to the best of our knowledge, existing work mainly focuses on injecting noises into the encoding sequence, while the feasibility of injecting noises into the decoding sequence for plms remains an open question. this is the primary interest of this work. on the other hand, recent researches @xcite @xcite @xcite @xcite demonstrate that enhancing plms with real-world knowledge is crucial for knowledge-driven downstream tasks, including entity typing, relation classification, sentiment analysis, and entity-related question answering. however, the pre-training objectives for the abovementioned studies are usually designed for natural language understanding (nlu) tasks. in this work, we focus on knowledge-oriented sequenceto-sequence (seq2seq) pre-training objectives for plms, so that they can be applied to natural language generation (nlg) tasks, such as the data-totext task. given a noise-corrupted sentence, such as "tom cruise was born in @xcite [mask] in the year 1962", the standard seq2seq pre-training would predict the masked text fragment (i.e., "new york city") token by token, while our proposed pretraining would generate the correct output by learning facts regarding the concerned entity. we argue that given merely "tom cruise was born in", if a plm can immediately predict the masked fragment to be "new york city", it suggests that the plm has learned the fact ("tom cruise", "born in", "new york city"). in contrast, if the plm could only predict "york" after being provided with the entity's partial ground-truth token "new", which is what most existing plms are capable of, although it indeed correctly predicts the masked tokens, it does not truly learn the fact or the entity of tom cruise's birthplace. thus, we propose prefix-masked decoding for knowledge enhanced sequence-tosequence pre-training (prince), which decodes entity tokens with noisy prefixes rather than groundtruth tokens. for example, when predicting "york", a mask symbol is fed into the decoder as the prefix, in place of "new". chen et al. (2020) that generates knowledge-enriched text based on a knowledge subgraph from wikidata, prince can directly incorporate entity knowledge into plms based on the raw text with extracted entities, alleviating error propagations throughout (a) mass decoder attention encoder ,-./01 2 2 /3 2 45-3 670 859 :16 /3 ;5-< ;5-< :16 ,/=> (b) prince decoder attention encoder ,-./01 2 2 /3 2 45-3 670 859 2 /3 2 ;5-< :16 ,/=> common token entity token figure 1: a pre-training framework comparison of mass and our proposed prince. the major difference is that prince predicts entity tokens without feeding the previous ground-truth entity token into the decoder. the additional data processing, such as entity alignment and knowledge triple retrieval. after pre-training prince, we evaluate it on two data-to-text datasets that require entity knowledge: webnlg @xcite and wik-ibio @xcite . on both datasets, our model achieves new state-of-the-art results. our main contributions are as follows: ## background we first introduce the existing denoising autoencoder schema for the seq2seq pre-training. given a sentence with a masked text fragment, mass @xcite proposes a masked seq2seq pre-training objective that is optimized to predict the masked tokens auto-regressively. recent work, such as bart @xcite and t5 @xcite , has exhibited gains by applying a wide range of approaches to inject noises into the input, such as sentence permutation, document rotation, and text infilling. in this work, we adopt the framework of mass. prince is simple and flexible that we anticipate it would be able to integrate to other denoising autoencoder schemas with more sophisticated noising approaches. next, we will describe the basic framework of the masked seq2seq pre-training. given a the loss function l t for each time t is the negative log likelihood of the token x @xmath0: the loss is: for the example shown in figure 1 , the masked entity fragment is {new, york, city}. when predicting "york", for mass, the previous groundtruth sequence "new" is fed into the decoder. while for prince, "new" is replaced by m. in this way, each token in the entity fragment is generated without any indication from other entity tokens. ## conclusion and future work we propose prince, a seq2seq pre-training model equipped with a knowledge-enhanced decoder that predicts entity tokens with masked prefixes. prince achieves new state-of-the-art results on two data-to-text datasets. in the future, we will adopt prince to other pre-training schemas and more knowledge-driven tasks. our code is publicly available foot_1 .