P 3 LM : Probabilistically Permuted Prophet Language Modeling for Generative Pre-Training (2022)

Abstract: Conventional autoregressive left-to-right (L2R) sequence generation faces two issues during decoding: limited to unidirectional target sequence modeling, and constrained on strong local dependencies.To address the aforementioned problem, we propose P3LM, a probabilistically permuted prophet language model, which strengthens the modeling of bidirectional information and long token dependencies for sequence generation.Specifically, P3LM learns to generate tokens in permuted order upon an order-aware transformer decoder, as well as to generate the corresponding future N tokens with a multi-stream attention mechanism.Extensive experiments are conducted on the GLGE benchmark, which includes four datasets for summarization, two for question generation, one for conversational question answering, and one for dialog response generation, where P3LM achieves state-of-the-art results compared with strong publicly available generative pre-training methods.

## introduction natural language generation (nlg), aiming to automatically generate a sequence of tokens, are widely explored on tasks such as summarization, question answering and generation, dialog response generation, and machine translation. recently, generative pre-training models @xcite @xcite @xcite @xcite @xcite , which accumulate knowledge based on large-scale unsupervised conditional language modeling, have achieved remarkable improvements on downstream nlg tasks compared with conventional methods. a typical generative pre-training model @xcite follows the transformer @xcite , prophet decoding: y t is predicted based on @xmath0, or @xmath1. p foot_1 lm decoding: y is autoregressively decoded in terms of order z, where y zt is predicted based on y @xmath2, or y @xmath3. y 0 = ⟨s⟩ is the start of a sentence. in order to enable a language model to learn bidirectional context information, auto-encoding ones, such as bert @xcite known as a masked language model (mlm), are pre-trained based on randomly masked token prediction. in addition, autoregressive ones, such as xlnet foot_0 @xcite known as a permutation language model (plm), are designed to reconstruct a partial sequence in permuted order. however, directly applying these methods on language generation is not feasible, since they are designed for natural language understanding (nlu), which are usually handled by just one encoder or decoder @xcite . to prevent overfitting on strong local dependencies during decoding, prophetnet @xcite is proposed to predict n future tokens. however, the future token prediction strategy predicts at most n (typically @xmath4, which has limited ability on long dependency modeling. besides, due to the l2r decoding, the unidirectional target context modeling issue still exists. to further enhance the ability of long dependency modeling, as well as capturing bidirectional information of target sequences, we propose p 3 lm, a probabilistically permuted prophet language model. p 3 lm learns to generate tokens in permuted order with an order-aware transformer decoder, as well as predicting the corresponding n future tokens with a multi-stream attention mechanism. figure 1 illustrates the idea of the proposed p 3 lm. for instance, given a target sequence @xmath5, y 2 , y 3 ]=@xmath6, 1, 3], p 3 lm learns to generate sequence y in order z, i.e., @xmath7. meanwhile, it also learns to predict future tokens in terms of z, e.g., predicting sequence as the future token of order at time step @xmath8. the above design makes p 3 lm capable of capturing bidirectional information of target sequence, and strengths the modelling of long dependencies. extensive experiments are conducted on the glge @xcite benchmark, a general language generation evaluation benchmark consisting of four datasets for summarization, two for question generation, one for conversational question answering, and one for dialog response generation, where our proposed p 3 lm achieves 0.9 absolute and 2.5% relative improvements on the overall score compared with the public available state-of-the-art model, i.e., prophetnet. to conclude, the contributions are as follows: (i) we propose p 3 lm, a permutation over prophet decoding net, for generative pre-training, which utilizes bidirectional context information and enhances long token dependency modeling on target sequences; (ii) we conduct extensive experiments on downstream language generation tasks and show that p 3 lm obtains new state-of-the-art results on glge benchmark compared with published methods; (iii) three p 3 lm models, which cost about 100,000 dollars, are pre-trained based on large scale datasets and will be released for further research on generative pre-training and language generation for the nlp community. models structure tasks features during decoding order longdep bidir bert enc nlu ---roberta enc nlu ---xlnet enc nlu ---electra enc nlu ---albert enc nlu ---gpt dec nlu&nlg l2r shallow no unilm enc/dec nlu&nlg l2r&r2l shallow shallow t5 enc-dec nlu&nlg l2r shallow no bart enc-dec nlu&nlg l2r shallow no pegasus enc-dec nlg l2r shallow no palm enc-dec nlg l2r shallow no mass enc-dec nlg l2r shallow no prophetnet enc-dec nlg l2r medium no p 3 lm enc-dec nlg permuted strong strong ## related work typical pre-trained language models are shown in ## experiment extensive experiments are conducted. in section 4.1, pre-training details of p 3 lm are introduced. in section 4.2, we show that p ## conclusion a probabilistically permuted prophet language modeling, p 3 lm, is proposed for generative pre-training. p 3 lm models sequences by considering both left-to-right and random permutation orders, equipped with a prophet mechanism for future token prediction. extensive experiments are conducted on glge, a general natural language generation evaluation benchmark, where p 3 lm achieves state-of-the-art results compared with public available generative pre-training methods.