Z -Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (2023)

Abstract: This paper presents Z-Code++, a new pre-trained language model optimized for abstractive text summarization. The model extends the state-of-the-art encoder-decoder model using three techniques. First, we use a two-phase pre-training to improve the model’s performance on low-resource summarization tasks. The model is first pre-trained using text corpora for language understanding, then is continually pre-trained on summarization corpora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierarchical manner. Z-Code++ createsa new state-of-the-art on 9 of 13 text summarization tasks across 5 languages. Our model is parameter-efficient in that it outperforms the 600x larger PaLM540B on XSum, and the finetuned 200x larger GPT3175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models.

## introduction text summarization aims at producing a concise and fluent summary while preserving salient content and overall meaning of the source documents. it has been applied in a wide range of real-world applications, e.g., summarizing web search results for interactive information retrieval @xcite and generating medical summaries from doctor-patient conversation transcripts @xcite . while the extractive approach is the dominant approach in commercial systems due to its simplicity and effectiveness @xcite , the abstractive approach is getting more attention in the research community as neural language models are used (e.g., @xcite @xcite @xcite . compared to the extractive approach where a summary is constructed using extracted sentences, abstractive summarizers paraphrase the idea of the source documents in a new form, and have a potential of generating more concise and coherent summaries. however, good abstractive summarizers are harder to develop since we have to deal with problems like semantic representation, inference and low-resource text generation, which are more challenging than sentence extraction. recently, largescale pre-trained language models (plms) such as pegasus @xcite , gpt @xcite , t5 @xcite , have been applied for abstractive summarization. while these models can produce surprisingly fluent text, the generated summaries often contain factual inconsistencies, caused by distorted or fabricated facts about the source documents, which is known as the hallucination problem @xcite @xcite . in addition, since the amount of text in the source documents can be very large, it is expensive to train an end-to-end abstractive model (e.g., an encoder-decoder transformer model) given the memory constraints of current hardware and the latency constraints of applications such as online document summarization for interactive information retrieval. therefore, a two-stage approach is widely used, where a subset of document sentences is coarsely selected using an extractive summarizer, and an abstractive summarizer generates the summary conditioning on the extraction @xcite . this approach is sub-optimal in that salient information might be missed in the extraction. in this paper, we propose a new encoderdecoder plm optimized for abstractive summarization, z-code++, which significantly extends z-code @xcite , a state-of-the-art plm developed for machine translation, as follows. first, z-code++ is pre-trained on web text using two tasks, replaced token detection (rtd) and corrupted span prediction (csp). rtd uses a generator to generate ambiguous corruptions and a discriminator to distinguish the ambiguous tokens from the original inputs @xcite . rtd is proved to be more sample-efficient than the classic mask language modeling (mlm) task in learning text representations for language understanding @xcite . in csp, a consecutive segment of tokens are corrupted and the model is learned to predict the corrupted spans using all the uncorrupted tokens in the original input @xcite . csp can be viewed as a generalized form of gap sentences generation (gsg), a pre-training task tailored to abstractive summarization @xcite , where the spans are entire sentences. csp outperforms gsg in our experiments. in the second phase of grounded pre-training @xcite , the model is continually trained on summarization corpora of documents-summary pairs to better support low-resource fine-tuning to downstream summarization tasks that require the model to produce summaries grounded in source documents. we find in our experiments that grounded pre-training significantly boosts the results on downstream tasks in low-resource settings. to handle the large input documents, we use fusion-in-encoder (fie), a simple yet effective method of encoding long sequences in a hierarchical manner. it works by first splitting the input sequence into small chunks, applying attention on each chunk locally to get the chunk representation, and applying attention globally on the concatenated chunk representations to get the representation of the original input. in addition, we replace the self-attention layer in the encoder with the disentangled attention (da) layer @xcite , where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. da is motivated by the observation that the attention weight of a word pair depends on not only their contents but their relative positions. for example, the dependency between the words "deep" and "learning" is much stronger when they occur next to each other than when they occur in different sentences. we show in our experiments that da leads to a more effective abstractive summarizer. for evaluation, we have pre-trained two z-code++ models on english data and multi-lingual data, respectively. the english model is trained using 160g english text data and the vocabulary of debertav2 @xcite . the multilingual model is trained on mc4 corpus which is the same as mt5. these models are evaluated on 13 text summarization tasks across 5 languages, and create new state of the art on 9 tasks. as of may 6th, 2022, z-code++ sits atop of the xsum leaderboard, surpassing ul2 20b , t5 11b and pe-gasus. it is worth noting that our models are very parameter-efficient. for example, z-code++ outperforms palm 540b , which is 600x larger in model parameters, on xsum, and outperforms a fine-tuned, 200x larger, gpt3 175b on samsum. in zero-shot and few-shot settings, our models outperform more substantially the competing models. ## z-code++ this section describes three modeling techniques we have exploited to optimize z-code++ for abstractive summarization, including two-phase pretraining, disentangled attention, and long sequence encoding. ## conclusions we present z-code++, an efficient and effective pre-trained language model optimized for abstractive text summarization. the model extends the encoder-decoder model using three techniques. the first is a two-phase pre-training process, where the model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. the second technique is the use of the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively. the third is the fusion-in-encoder method for encoding long sequence inputs. we present a comprehensive empirical study to validate the effectiveness of z-code++. the model creates new state of the art on 9 out of 13 text summarization tasks across 5 languages. in addition, we show that our model is parameter-efficient in that it outperforms the 600x larger palm 540b on xsum, and the finetuned 200x larger gpt3 175b on samsum. z-code++ also generalizes well to low-resource downstream tasks. for example, in zero-shot and few-shot settings, our model outperforms more substantially the competing models. however, evaluation @xcite and hallucinations are still two long-standing problems of summarizations that we do not touch with in this work, in the future we will 1) explore evaluation metrics that correlate well with human experience, 2) learn to summarize to better align with human preferences @xcite , and 3) ground summarization models on world knowledge to largely reduce hallucinations @xcite .