Abstractive Text Summarization Using the BRIO Training Paradigm (2023)

Abstract: Summary sentences produced by abstractive summarization models may be coherent and comprehensive, but they lack control and rely heavily on reference summaries. The BRIO training paradigm assumes a non-deterministic distribution to reduce the model’s dependence on reference summaries, and improve model performance during inference. This paper presents a straightforward but effective technique to improve abstractive summaries by fine-tuning pre-trained language models, and training them with the BRIO paradigm. We build a text summarization dataset for Vietnamese, called VieSum. We perform experiments with abstractive summarization models trained with the BRIO paradigm on the CNNDM and the VieSum datasets. The results show that the models, trained on basic hardware, outperform all existing abstractive summarization models, especially for Vietnamese.

## introduction text summarization reduces the size of the original text while preserving its main content. the two main approaches for constructing summaries are extractive and abstractive. extractive summarization directly lifts sentences or words which convey key topics of the original documents, and concatenates them. abstractive summarization discovers the primary content of the documents and generates summaries. abstractive summaries are usually more natural and coherent than extractive summaries. most abstractive summarization models follow the encoder-decoder framework. existing abstractive summarization models are trained using maximum likelihood estimation and rely on the reference summaries. @xcite propose a brio training paradigm to address reliance on reference summaries by assuming non-deterministic distribution of system-generated candidate summaries. in this paper, we use the brio training paradigm for abstractive summarization models to construct summaries for documents in english and vietnamese. we make the following contributions: the remainder of this paper is organized as follows. related work is presented in section 2. section 3 introduces a large dataset for summarization in vietnamese, named viesum. experiments and discussion are presented in section 4. section 5 concludes the paper. sheng et al. (2022)'s siamese semantic preserving generative adversarial net (sspgan) uses a transformer-based generator to generate summaries. a siamese transformer-based discriminator captures the semantic consistency between the source document and the corresponding summary. during adversarial training, the discriminator calculates a reward for each word generated. on the gigaword dataset, sspgan model achieves better results than many existing abstractive text summarization models such as deep recurrent generative decoder @xcite , actor-critic approaches from reinforcement learning @xcite , and transformer @xcite . @xcite develop the pagesum model for abstractive summarization by incorporating locality bias in both encoder and decoder. each document is partitioned into non-overlapping pages. the encoder, which is an abstractive summarizer, encodes each page and makes local predictions. the decoder predicts output based on a weighted combination of local predictions. the authors fine-tune the bart model @xcite for abstractive summarization and investigate several approaches to locality, such as spatial locality, discourse locality, and document locality. page-sum outperforms abstractive summarization models such as longformer encoder-decoder @xcite , encoder-decoder attention with headwise positional strides @xcite , and bart with hierarchical attention transformer @xcite . however, pagesum takes a long time to train, requires large memory size, and fails to capture long distance dependencies. several studies use pre-trained models for abstractive text summarization. @xcite use mt5 @xcite and sequence to sequence parsbert @xcite to construct abstractive summaries for persian texts. t5 @xcite and bert @xcite have also been used to construct abstractive summaries @xcite . @xcite summarize covid-19 biomedical research articles using bert and gpt-2 @xcite . features of documents are extracted and integrated into an abstractive model to improve summary generation. nambiar et al. (2022) develop an encoder-decoder model using attention, in which pos features are incorporated to the word embedding layers to enhance the word vectors. experiments on a dataset in malayalam show that the integration of attention model and pos features is better than the seq2seq and attention models. barna and heickal (2021) adapt the pointer generator network for abstractive summarization by combining a pre-trained word embedding layer for transferring semantic similarity and topic features for better topic coverage. a drawback of usual abstractive summarization is the omission of named entities. to ameliorate, @xcite train a named entity recognition model based on roberta to discover named entities. then, the bart masked named entity language model is trained to pay attention on the name entities. finally, bart is fine-tuned for text summarization. most studies to construct abstractive summaries in vietnamese use an encoder-decoder framework or a pre-trained model. @xcite integrate sentence positions and term frequencies into a pointer generator network with a coverage mechanism to perform the abstractive summarization for vietnamese documents. @xcite construct abstractive summaries for online newspapers using rnn with attention, bilstm with copy generator, standard transformer, bert, and sequence-to-sequence abstractive models using bottom-up approach. @xcite perform experiments to summarize vietnamese documents using transformer-based encoder-decoder architectures such as transformer, @xcite , and vit5 @xcite . ## viesum dataset we construct a viesum dataset for vietnamese consisting of 1,627,415 documents and their corresponding summaries, grouped into 23 categories. in particular, beautifulsoup foot_0 and newspaper3k foot_1 are used to collect and extract articles from popular online newspapers in vietnamese such as vnexpress.net, dantri.com.vn, danviet.vn, vietnamnet.vn, laodong.vn, and vov.vn. the summaries and content documents are considered reference summaries and documents, respectively. ## experimental results we perform experiments in the google colaboratory environment, nvidia tesla t4 16gb. we use the cnndm foot_2 dataset in english, and our viesum dataset in vietnamese. due to limitation of the hardware, we perform experiments with 70,000 documents picked randomly and their corresponding reference summaries from viesum. each dataset is split into 3 parts including 75% for training, 8% for validation, and 17% for testing. in this paper, the pre-trained bart 512-lengthbased and t5 512-length -based models are used as backbones for generating abstractive summaries. the bart @xcite and t5 @xcite ) models are trained on the cnndm dataset, while the bartpho @xcite and vit5 @xcite are trained on the viesum dataset. all models are base models. to make it easy for comparison, we use the same parameters as suggested by the original authors. ## conclusion we investigated abstractive summarization models trained with the brio paradigm. experiments show that we can improve abstractive summarization models by fine-tuning the backbones before training them with brio. in particular, the summarization models trained with brio outperform other summarization models in vietnamese. we also discuss issues with the brio paradigm for further exploration. in addition, we built the viesum dataset for summarization in vietnamese. for future work, we will ask volunteers to evaluate and provide feedback on a small subset of the viesum dataset.