S ent BS : Sentence-level Beam Search for Controllable Summarization (2022)

Abstract: A wide range of control perspectives have been explored in controllable text generation. Structure-controlled summarization is recently proposed as a useful and interesting research direction. However, current structure-controlling methods have limited effectiveness in enforcing the desired structure. To address this limitation, we propose a sentence-level beam search generation method (SentBS), where evaluation is conducted throughout the generation process to select suitable sentences for subsequent generations. We experiment with different combinations of decoding methods to be used as sub-components by SentBS and evaluate results on the structure-controlled dataset MReD. Experiments show that all explored combinations for SentBS can improve the agreement between the generated text and the desired structure, with the best method significantly reducing the structural discrepancies suffered by the existing model, by approximately 68%.

## introduction controllable text generation is receiving increasing attention due to its wide range of applications. depending on the use cases, the controllable generation tasks may focus on a wide range of control perspectives, such as entities @xcite , aspects @xcite , and keywords @xcite . recently, @xcite propose a sentencelevel labeled meta-review dataset, mred, for the controllable summarization task from a new control perspective that focuses on controlling the structure of the output summary. the input consists of several reviews on the same research paper, and a control sequence specifying the desired summary structure. for instance, with a control sequence of "abstract | strength | decision", the generated output should be composed of a sentence that summarizes the contents of the paper, followed by a sentence discussing the strengths, then the last sentence giving the final decision. previous controllable summarization models are commonly fine-tuned on pre-trained transformer architectures @xcite such as bart @xcite and pegasus @xcite , with the control signals merged into the text input or prompts @xcite @xcite @xcite . previous works mainly focus on improving the summary's similarity with the gold reference, leaving room for further improvement on the controllability. in particular, the bestperforming model on the recently released mred dataset still generates around 29% of the sentences that do not follow the control structure foot_0 , which is far from satisfactory. in this paper, we explore how to enhance the structure-controllability in summarization. specifically, we notice the following possible pitfalls in the existing summarization models. first, those models usually treat generation as a standalone process, which continuously generates the tokens solely based on the logits predictions, without stopping to reconsider whether the generated sequences satisfy the control signals. moreover, autoregressive models can suffer from error propagation in generation due to self-attention @xcite . therefore, if the previous sequences are not well-controlled, subsequent generations may deviate further from the desired output. motivated by this, we propose the sentence-level beam search (sentbs) method to address the identified issues. duces multiple sentence options, evaluates and selects the best sentence according to both the control structure as well as the model's log-likelihood, then continues the generation for the next sentence. experiments show that sentbs can significantly improve the model's structure-controllability. in particular, our best setting removes up to 68% of control mistakes produced by the existing model on mred without compromising the summarization quality. the human evaluation further proves that sentbs significantly improves the fluency of summaries. to summarize, our main contributions are: (1) to the best of our knowledge, we are the first to conduct sentence-by-sentence controlled generation for structure-controllable summarization. (2) we propose sentbs, which conducts text generation with continuous evaluation on the sentence level with respect to the control requirements. this method can be easily applied to existing autoregressive models. (3) experiments show that sentbs significantly increases the model's structure controllability while preserving the summarization quality. decoding methods. common decoding methods include beam search @xcite @xcite , nucleus sampling @xcite , and beam sampling @xcite (see more details in appendix a). in our sentbs method, we use these methods foot_1 as sub-components for generating multiple sentence options. @xcite , @xcite and @xcite also works on decoding methods that enforce linguistic constraints on the generation structure. however, none of these works use a sentence-by-sentence generation and evaluation strategy based on whether the semantic content of the generation meets the control requirements. ## sentence-level beam search given the nature of autoregressive models, errors from previously generated tokens at inference time can be easily propagated to affect the subsequent tokens. to better control the structure, we propose sentbs, to evaluate and select the outputs at the sentence level during generation. sentbs can be easily applied to existing generation models during inference only. in this section, we explain how sentbs leverages the control sequence where each control label corresponds to the desired sentence category. please see more details on the task and dataset in section 4.1. ## conclusions in this work, we refocus the attention of controllable summarization from the similarity with gold toward the model's controllability. we propose a simple and effective method, sentbs, to enhance the structure-controllability of the transformers bart model trained on mred. with extensive experiments and human evaluation, we show that without the need for retraining, sentbs can significantly improve the controllability of summary structure while achieving good generation quality.