ARMAN : P re-training with S emantically S electing and R eordering of S entences for P ersian A bstractive S ummarization (2021)

Abstract: Abstractive text summarization is one of the areas influenced by the emergence of pre-trained language models. Current pre-training works in abstractive summarization give more points to the summaries with more words in common with the main text and pay less attention to the semantic similarity between generated sentences and the original document. We propose ARMAN, a Transformer-based encoder-decoder model pre-trained with three novel objectives to address this issue. In ARMAN, salient sentences from a document are selected according to a modified semantic score to be masked and form a pseudo summary. To summarize more accurately and similar to human writing patterns, we applied modified sentence reordering. We evaluated our proposed models on six downstream Persian summarization tasks. Experimental results show that our proposed model achieves state-of-the-art performance on all six summarization tasks measured by ROUGE and BERTScore. Our models also outperform prior works in textual entailment, question paraphrasing, and multiple choice question answering. Finally, we established a human evaluation and show that using the semantic score significantly improves summarization results.

## introduction abstractive text summarization is the task of generating a short, fluent, and concise text that contains novel words and phrases other than the original document, preserving the primary subjects in the document. in contrast with extractive summarization, which aims to select the most important parts of the text to generate a summary, in abstractive summarization, the main goal is to generate a new persuasive piece of text as the summary of a document. earlier abstractive summarization works @xcite @xcite focused on training with large datasets containing pairs of documents and summaries in a supervised manner. by introducing transformer @xcite architecture and pre-training objectives and their positive impact on most nlp tasks, most current state-of-the-art (sota) methods focused on self-supervised objectives for pretraining transformer architecture in abstractive summarization tasks @xcite @xcite . however, current pre-training works give more points to the summary with more words in common with the main text and pay less attention to the semantic similarity between generated sentences and the original document. according to @xcite , the persian language is one of the top 25 spoken languages in the world. however, there are limited research studies in persian document summarization, and most of the prior works were mainly focused on extractive summarization. the main focus of this work is on persian abstractive summarization. nevertheless, our proposed method is language-independent. in this work, we first bring semantic similarity scores into a sentence selection schema to create a document's pseudo summary. briefly, we prepare a summary corresponding to each document in a dataset by selecting important sentences based on semantic scores in a self-supervised manner. next, we propose three novel objectives for pre-training a seq2seq transformer. our model, arman, uses transformer encoder-decoder structure and introduces a new combination of masking sentences with sentence shuffling and reordering objectives. we fine-tuned the models on six downstream tasks. according to an experiment, we found that letting the training model to copy pieces of the input text into the output summary does not lead to better results in downstream tasks. experiment results showed that our proposed models obtained sota performance in all persian abstractive sum-marization datasets on both rouge @xcite and bertscore @xcite . our models generated even better summaries than previous sota in zero and few shot settings when fine-tuned with a small number of document-summary pairs. we achieved sota results on two datasets with only 1k examples. moreover, our proposed models performed well in other nlu tasks, including textual entailment, question paraphrasing, and multiple choice question answering. finally, to ensure the significant improvement in summarization, we held a human evaluation, and we performed a student t-test on its results. the main contributions of this paper are threefold: ## related work automatic text summarization was mainly performed based on statistical methods @xcite ; most of them were striving to rank sentences by extracting their features @xcite @xcite . by rising of sequence to sequence learning with neural networks @xcite and attention mechanism @xcite usage in abstractive summarization tasks @xcite , a new era in abstractive summarization began. by introducing transformer @xcite and masked language modeling (mlm) methods of bert @xcite , most nlp tasks achieved a vast improvement gain using these pre-training methods and architectures. following bert's approach, many other language models were trained @xcite with differences in the amount of data used for pre-training and some optimizations on bert's pre-training method; most of them were only encoders. furthermore, encoder-decoder models were trained with a mixture of pre-training tasks; t5 @xcite and @xcite are two of them. since the pre-training of transformer was successful on most nlp tasks, some models were pre-trained for specific duties; pegasus @xcite ) is a pre-trained model that was trained specifically for summarization on c4 and huge-news corpora. pegasus trained with gap sentence generation (gsg) that masks the most important sentences based on syntactic similarity of sentences of a document. arman is different from pegasus in that we mask the most important sentences based on the semantic similarity of sentences. furthermore, we use only a single mask token for any consecutive sentences that should be masked. this approach helps the model learn how many sentences should be generated for each masked token in the input sequence. step @xcite is another pre-trained summarization model trained with mlm, next sentence generation (nsg), and sentence reordering (sr) objectives. arman uses sr as one of the pre-training methods in a modified form; we change the order of sentences in the input document. the model should select the most important sentences using semantic similarity of sentences to the document, then reorder them in the actual order that they appeared in the original document. in the persian language, some extractive summarization methods exist @xcite @xcite , but to the best of our knowledge, we know just one model on abstractive summarization. @xcite have used parsbert @xcite ) checkpoint with @xcite 's method to train a new sequence to sequence model with pre-trained weights for the encoder and decoder. in this regard, arman is one of the first works on abstractive summarization for the persian language. also, arman was able to achieve sota results on all available datasets. ## methodology this section introduces a sentence selection method based on semantic similarity scores to make a pseudo summary. then, we propose three novel objectives for pre-training a seq2seq model for the abstractive summarization tasks. ## data collection this section introduces the datasets used for pretraining and fine-tuning models and the procedure of cleaning corpora. ## experiments in this section, we compare arman with previous works and conduct several experiments to assess the performance of the proposed methods. the codes for pre-training and fine-tuning of all models are publicly available on github foot_6 .