Boosting Summarization with Normalizing Flows and Aggressive Training (2023)

Abstract: This paper presents FlowSUM, a normalizing flows-based variational encoder-decoder framework for Transformer-based summarization. Our approach tackles two primary challenges in variational summarization: insufficient semantic information in latent representations and posterior collapse during training. To address these challenges, we employ normalizing flows to enable flexible latent posterior modeling, and we propose a controlled alternate aggressive training (CAAT) strategy with an improved gate mechanism. Experimental results show that FlowSUM significantly enhances the quality of generated summaries and unleashes the potential for knowledge distillation with minimal impact on inference time. Furthermore, we investigate the issue of posterior collapse in normalizing flows and analyze how the summary quality is affected by the training strategy, gate initialization, and the type and number of normalizing flows used, offering valuable insights for future research.

## introduction abstractive summarization @xcite @xcite aims to generate summaries by rephrasing or introducing novel words to capture the most salient information in the source text. many abstractive summarization models @xcite @xcite are based on the transformers architecture @xcite and have consistently produced state-of-theart summarization quality. however, issues such as exposure bias @xcite , lack of text generation diversity @xcite , and insufficient capturing of semantic information @xcite remain. variational models have gained increasing research interest @xcite @xcite as they address these issues by introducing uncertainty in predictions through learning a probability distribution over latent variables. a variational model enables diverse text generation @xcite , smoother output spaces, and semantically meaningful latent codes @xcite that guide the generation of coherent and informative summaries. nonetheless, existing variational models have not fully achieved the aforementioned desirable properties due to two main challenges. firstly, the semantic information in the source text may possess a complex structure. however, since introducing latent variables complicates parameter estimation, many current models @xcite represent latent codes using a gaussian distribution, which is insufficient for capturing the intricacies of the latent space and could potentially reduce model performance. to enrich latent distributions, researchers suggest replacing the highly restricted isotropic gaussian with normalizing flows @xcite . normalizing flows can generate complex distributions while preserving density in an analytical form, and they have been integrated into variational autoencoder (vae) @xcite and variational encoder-decoder (ved) @xcite frameworks to better approximate the latent posterior. this approach has found application in various domains, including text generation @xcite , neural machine translation @xcite , and dialogue generation @xcite . despite this progress, the operating characteristics of normalizing flows on summarization tasks have yet to be investigated. secondly, as reported by previous studies @xcite @xcite , variational models tend to experience posterior collapse during training, which occurs when the kl term vanishes to zero, indicating that the model fails to learn meaningful latent codes. this problem becomes more severe when modeling discrete data with a strong auto-regressive decoder @xcite , which is the case for transformerbased summarization models. to resolve this issue, several solutions have been proposed, such as employing a less auto-regressive decoder network @xcite @xcite , modifying the training objective @xcite @xcite , and proposing new training strategies @xcite . however, most existing work focuses on the vae framework with gaussian latent distribution, yet limited work considers the ved framework with normalizing flows. in particular, two questions remain unclear: (1) when the latent distribution is modeled by normalizing flows, does the posterior collapse problem still exist? (2) when posterior collapse exists, what are the appropriate strategies to achieve good summarization quality within the ved framework? this paper introduces flowsum foot_0 , a normalizing flows-based ved framework for transformerbased summarization, along with a controlled alternate aggressive training (caat) strategy and a refined gate mechanism to resolve the two challenging issues. our contributions include: this article consists of five sections. section 2 provides an overview of normalizing flows, ved, and a summary of related studies. section 3 describes the proposed model architecture and the training strategies employed. section 4 presents the experimental setup and results, and section 5 concludes the paper with some discussions. ## conclusions and discussions this paper introduces flowsum, a normalizing flows-based variational encoder-decoder (ved) framework for text summarization. it outperforms a leading non-latent model across multiple datasets. this enhanced performance is attributed to the flexible posterior distributions provided by normalizing flows. we also analyze the operating characteristics and the posterior collapse problem of normalizing flows and propose an effective training strategy for complex flows. moreover, we demonstrate that in-corporating normalizing flows is highly effective for knowledge distillation with minimal impact on inference time. flowsum illustrates the advantages of incorporating flexible latent modeling. considering the remarkable achievements of latent diffusion models (ldms) in generating images @xcite , adopting ldms for capturing latent representation may produce comparable or even superior outcomes in text summarization. in this scenario, the gating mechanism may not be an appropriate choice. a direct correlation between the latent vector and the target text may be more suitable for executing the diffusion process. enhancing the architecture to leverage diffusion models could be a potential avenue for future research.