Semantic-Preserving Abstractive Text Summarization with S iamese Generative Adversarial Net (2022)

Abstract: We propose a novel siamese generative adversarial net for abstractive text summarization (SSPGAN), which can preserve the main semantics of the source text. Different from previous generative adversarial net based methods, SSPGAN is equipped with a siamese semantic-preserving discriminator, which can not only be trained to discriminate the machine-generated summaries from the human-summarized ones, but also ensure the semantic consistency between the source text and target summary. As a consequence of the min-max game between the generator and the siamese semantic-preserving discriminator, the generator can generate a summary that conveys the key content of the source text more accurately. Extensive experiments on several text summarization benchmarks in different languages demonstrate that the proposed model can achieve significant improvements over the state-of-the-art methods.

## related work automatic text summarization can be broadly di vided into extractive and abstractive summarization. the extractive methods simply extract important parts of the source text and reorganize them in a certain order @xcite @xcite . in comparison, abstractive text summarization is closer in princi ple to the process of manual summarization, which extracts the essential information of the source text and describes it in a shorter version as the abstrac tive summary. in this paper, we focus on abstractive text summarization. previous works on abstractive text summariza tion are mainly designed with statistical methods and rule-based methods @xcite @xcite . recently, the sequence-to-sequence neu ral framework becomes predominant on the task of abstractive text summarization @xcite @xcite . later on, with the advent of transformer @xcite , more and more works choose it as the base model in their frameworks. for the abstractive text summarization task, out-of-vocabulary (oov), repetitions and lack of saliency are three dominant challenges. to tackle the problem of oov, some works introduce the pointer network and copy mechanism @xcite @xcite . on the issue of repetitions, @xcite adopts a coverage mechanism, which is in spired by the coverage vector from neural machine translation @xcite . regarding saliency, some works @xcite focus on how to optimize the attention mechanism, while @xcite tries to enhance the fac tual consistency with a fact corrector. meanwhile, @xcite adopts the content planning to improve the performance of abstractive summa rization model. however, the essence of saliency, which is the sentence-level semantic consistency between the source text and the generated summary, is intuitive yet usually overlooked. the proposed training principle is based on ad versarial learning @xcite . in conventional adversarial training, a generator and a discriminator compete with each other, forcing the generator to produce high quality samples that can fool the discriminator. adversarial training typically excels in image generation @xcite , with less applications in natural lan guage processing tasks @xcite , mainly due to the difficulty of propagating the signals from the discriminator to the generator through the discretely generated tokens. @xcite addresses this issue with a reinforcement learning approach for sequence generation. thus, the adversarial training paradigm can improve the model on the sentence-level instead of the vanilla token-level (e.g., maximum likelihood estimation). to address the semantic inconsistency problem mentioned above, we introduce the paradigm of siamese net into gans. siamese net is a class of neural network architectures that contain more than one identical or different sub networks, which depends on whether the inputs are similar or not. siamese net is generally used to measure the simi larity between the inputs by comparing their corre sponding output feature vectors, and can be broadly divided into two types: true siamese net and pseudo siamese net. the true siamese net contains identi cal sub networks which share the same architecture and network parameters, while the pseudo siamese net contains sub networks which have different pa rameters and even different architectures. among the existing works, @xcite is the first to adopt siamese net into unsupervised sentence embedding learning. @xcite proposes malstm to learn sentence simi larities with manhattan distance. @xcite considers similarity matching of a sentence pair as a binary classification task and replaces the manhattan distance with cosine similarity. re cently, @xcite introduces the principle of siamese net to fine-tune bert @xcite for better sentence embedding. different from previous gan-based abstractive text summarization model in the work of @xcite , by incorporating siamese net into gans, the generator can generate summaries which are more semantically consistent with the source texts. as far as we know, this work is the first attempt to apply siamese net to the gan-based sequence-to sequence generation task. ## siamese semantic-preserving generative adversarial net in this section, we introduce the architecture of the proposed siamese semantic-preserving gen erative adversarial net (sspgan) in detail. the model consists of two main components. the first component is a standard transformer-based sum mary generator g (figure 2 ). during adversarial training, the generator g is treated as an agent tak n n figure 2: the summary generator, taking a conven tional transformer based encoder-decoder architec ture @xcite , where the predicted word from the previous step serves as the input of the current step during inference. we omit some layers for brevity. in the following, we describe the generator g and the siamese semantic-preserving discriminator d in detail. ## conclusion this paper presents a novel siamese generative ad versarial net (sspgan) which can preserve the se mantic consistency between the source text and the target summary for abstractive text summarization. in sspgan, a novel semantic similarity based re ward is introduced to further augment the ganbased abstractive text summarization to preserve the semantic consistency and convey the key con tent in the source text. it is worth noting that ssp gan addresses the problem of saliency for text summarization from a totally different perspective of semantic consistency, therefore it is orthogonal to some state-of-the-art methods which focus on attention mechanism, and can be applied to them for further improvements. the inverse square root learning rate decay is ap plied for initial warm up and annealing with 4000 steps. for the discriminator, we adopt rmsprop optimizer with the learning rate of 0.0005 and η = 0.7. the dropout rate is set to 0.3 for both mod els. during adversarial training, for both models, the learning rate is set to 0.00001 without changing the optimizer. k in monte carlo rollout is set as 20 and λ is 0.7. in the proposed architecture, there are 2 hyperparameters η and λ need to be jointly tuned during training. here we conduct a grid search to find a proper combination of these hyper-parameters. for both η and λ, the value is selected in set [0.1, 0.3, 0.5, 0.7, 0.9] and we experimentally find that the η of 0.7 and the λ of 0.7 give the best results on validation sets. a.2 pseudo code algorithm 1 siamese semantic-preserving gan require: generator g θ , siamese semanticpreserving discriminator d ϕ , a text summa rization dataset @xmath0, ŷ) 1: initialize g θ , d ϕ with random weights θ, ϕ 2: pre-train g θ using (1) on s 3: generate negative summaries y with g θ for training d 4: pre-train d ϕ using (8) on the combination of (x, y) and s 5: while g θ not converged do 6: for g-steps do 7: generate a sequence @xmath1, • • • , y t ) ∼ g θ 8: for t in 1 : t do g θ 9: calculate r d ϕ (y 1:t-1 , x, y t ) using (12) 10: end for 11: update generator with policy gradient (13) 12: end for 13: for d-steps do 14: generate negative pairs (x, y) using latest g θ and combine them with given positive pairs s 15: train discriminator d ϕ by (8) 16: end for 17: end while