H ydra S um: Disentangling Style Features in Text Summarization with Multi-Decoder Models (2022)

Abstract: Summarization systems make numerous “decisions” about summary properties during inference, e.g. degree of copying, specificity and length of outputs, etc. However, these are implicitly encoded within model parameters and specific styles cannot be enforced. To address this, we introduce HydraSum, a new summarization architecture that extends the single decoder framework of current models to a mixture-of-experts version with multiple decoders. We show that HydraSum’s multiple decoders automatically learn contrasting summary styles when trained under the standard training objective without any extra supervision. Through experiments on three summarization datasets (CNN, Newsroom and XSum), we show that HydraSum provides a simple mechanism to obtain stylistically-diverse summaries by sampling from either individual decoders or their mixtures, outperforming baseline models. Finally, we demonstrate that a small modification to the gating strategy during training can enforce an even stricter style partitioning, e.g. high- vs low-abstractiveness or high- vs low-specificity, allowing users to sample from a larger area in the generation space and vary summary styles along multiple dimensions.

## introduction abstractive summarization @xcite involves a combination of generation decisions, such as what content to directly copy from the input and what content to paraphrase, the level of specificity vs generality, length, readability, etc. of generated summaries. current summarization systems @xcite implicitly encode these decisions in their parameters, but provide no mechanism for end users to specify their stylistic preferences. commonly used decoding methods such as beam search, top-k de-coding @xcite or diverse decoding @xcite tend to generate stylistically similar outputs, and cannot be queried for multiple diverse summaries without sacrificing quality. prior work in style transfer @xcite target styles that are not relevant to summarization (e.g. sentiment, shakespearean language, etc.) and use explicit interventions to enforce style. instead, we ask: what style combinations naturally occur in abstractive summarization datasets and can models automatically disentangle them? in this paper, we propose hydrasum -a new summarization architecture that disentangles the different stylistic decisions made by abstractive summarization models from the models weights into an explicit model component. our model contains a single transformer-based encoder to encode the input document and a mixture-of-experts with multiple decoders for summary generation. at each time step of the generation phase, the next token's probability distribution is computed by combining the output probabilities of all individual decoders. this allows our model to distribute the diverse stylistic and lexical features encountered in the training data, even those within the same reference summary, across the parameters of separate decoders. during inference, we leverage the modularity in the decoder framework to sample from these individual decoders, each of which generates stylistically-distinct summaries. as a toy example, consider a 2-decoder scenario in which one decoder learns to only copy phrases or words from the input document, while the second decoder only learns paraphrasing and syntactic transformations. while individual decoders cannot cover the range of stylistic variations in the dataset, a weighted combination or mixture of the two decoders can be used to model the summarization dataset. in practice, we found that this partitioning of summarization "skills" between decoders input article: insights into the workings of the human body that leonardo da vinci could only obtain by dissecting scores of corpses and recording the results in exquisite drawings will be displayed for the first time beside modern 3d films, ct and mri scans, which show how close the renaissance genius got to the truth of what lies under the skin. @xcite the edinburgh show will be the first to compare leonardo's results with scalpel and pen with the best results of modern technology. […] the exhibition will show how close leonardo got in some of his last medical experiments to discovering the role of the beating heart in the circulation of the blood, a century before william harvey worked it out. @xcite edinburgh show will be first to compare renaissance genius's results with best results of modern technology. edinburgh show will be first to compare renaissance genius's results with the best results of modern technology. edinburgh show will be first to compare leonardo's results with the best results of modern technology. ## methodology current state-of-the-art summarization models (e.g. bart, pegasus) use transformer-based encoder- figure 2: our proposed hydrasum architecture. the decoder network of standard models is modified to incorporate multiple decoders. the lower layers of these decoders have shared parameters and a gating mechanism is used to combine their output probabilities in a mixture-of-experts formulation. multi-decoder architecture let m be the total number of decoder blocks in a single decoder: e.g. @xmath0. in hydrasum, the parameters of the m(< m ) bottom layers are shared between the k decoders. this reduces the number of extra parameters introduced into the model architecture. the top mm layers of the different decoders are independently trained. the right block of figure 2 shows a detailed view of the multi-decoder architecture at a single time step i. ## experiments we conduct experiments on three news summarization datasets: cnn @xcite , newsroom foot_1 @xcite and xsum @xcite . the reference summaries in these datasets exhibit a mutuallydistinct stylistic properties and help evaluate hy-drasum's capabilities under these distinct test conditions. for all experiments, bart-large @xcite is used as the model initialization: in a kdecoder variant of hydrasum, all k decoders are initialized with the weights of bart-large's decoder. the weights of the gating mechanism g are randomly initialized from a normal distribution n (0, 0.02). we set the number of shared layers, i.e. m to 8, for all experiments. 3 our model architecture is implemented using the huggingface library @xcite . more training and inference details are in appendix a. we compare against the standard bart-based summarization baseline. for xsum, we use the publicly available bart-large-xsum checkpoint. for cnn and newsroom, we fine-tune the bart-large checkpoint on their corresponding training datasets ourselves. 4 beam decoding is used to generate summaries for all models. ## extreme partitioning in section 3, style partitioning was automatically driven by dataset properties. here, we investigate whether we can explicitly dictate which specific stylistic feature differs between two decoders. suppose our target feature (denoted by f ) is specificity: under this scenario, we want d0 to generate low-and d1 to generate high-specificity summaries. we should also be able to generate multiple mid-specificity summaries by mixing these two extreme decoders. in this section, we run experiments on two target features; abstractiveness (measured by 2-gram overlap) and specificity. our method to ensure d0 learns low-f and d1 learns high-f , we carefully control the contribution of each training example to individual decoder's training. our exact methodology is: (1) first, we pre-process the training data to derive their per-forget gold and oil. copper prices is the real winner this year. the red metal is up more than 20 percent from its late january low -and that's given one stock a big boost: freeport-mcmoran. the mining giant is up 40 percent in the same period, but one trader who relies heavily on the technicals and options market, is cautious on the stock, and he warned that the rally could be over. @xcite copper prices are up 20 percent this year, and one miner is up 40 percent. but one trader warns that the rally could be over. copper prices is the real winner this year. the red metal is up more than 20 percent from its late january low -and that 's given one stock a big boost. ## human evaluation following prior work @xcite , we conduct human evaluation to measure the quality of generated summaries. for 50 randomly sampled input articles from each dataset, we present mturk workers with 5 different generated summaries: baseline model summary, d0 and d1 summaries of the @xmath1. the workers were asked to rate each summary along 4 dimensions: relevance, coherence, grammatically and factuality. for the first 3, we ask for a rating on the 5-point likert scale. following @xcite , we seek binary labels (factual (1) or non-factual (0)) for factuality annotation. more details and task interface are in appendix c. we report the average score of all three annotations in ## related work prior work on style control in summarization focuses on features like length @xcite , abstractiveness @xcite , etc. it has also been studied for other generation tasks such as paraphrasing and story generation @xcite @xcite . these methods are over-specialized for the target style and cannot be easily generalized to more features. recently, gedi @xcite proposed using small lms as generative discriminators for specific attributes (e.g. toxicity) to guide the generation of larger models. similar class-conditional language models approaches (cc-lms) have been previously proposed @xcite to finetune models on specific attributes. contrary to these, hydrasum models can disentangle styles within the task-specific datasets without explicit style annotations, as well as cover the generation space between two 'extreme' styles. diverse generation has more widely been studied for other generation tasks, including decoding modifications @xcite , enforcing syntactic diversity @xcite , or through uninterpretable latent codes @xcite . in this work, we study diversity in style that naturally emerges under standard training and decoding. ## conclusion we propose a new summarization architecture hydrasum containing multiple decoders in a mixture-of-experts. our model automatically separates distinct summary styles, e.g. high or low abstractiveness, different levels of specificity, etc., across different decoders under the standard training regimen. we show that the proposed model is highly flexible; during inference, we can sample from either individual decoders or their mixtures to vary summary features.