MACS um: Controllable Summarization with Mixed Attributes (2023)

Abstract: Controllable summarization allows users to generate customized summaries with specified attributes. However, due to the lack of designated annotations of controlled summaries, existing work has to craft pseudo datasets by adapting generic summarization benchmarks. Furthermore, most research focuses on controlling single attributes individually (e.g., a short summary or a highly abstractive summary) rather than controlling a mix of attributes together (e.g., a short and highly abstractive summary). In this paper, we propose MACSum, the first human-annotated summarization dataset for controlling mixed attributes. It contains source texts from two domains, news articles and dialogues, with human-annotated summaries controlled by five designed attributes (Length, Extractiveness, Specificity, Topic, and Speaker). We propose two simple and effective parameter-efficient approaches for the new task of mixed controllable summarization based on hard prompt tuning and soft prefix tuning. Results and analysis demonstrate that hard prompt models yield the best performance on most metrics and human evaluations. However, mixed-attribute control is still challenging for summarization tasks. Our dataset and code are available at https://github.com/psunlpgroup/MACSum.

## the macsum dataset to provide a benchmark for controllable summarization, we propose macsum, a high-quality human-annotated mixed-attribute controlled summarization dataset. inspired by several previous studies on controllable generation @xcite @xcite , macsum is annotated with 5 types of control attributes, including topic, speaker, length, extractiveness, and specificity (section 3.1). as shown in figure 1 , these five attributes can be combined together in various designs (section 3.1). additionally, topic and speaker can have multiple values as well, i.e., more than one speaker or topic to focus on. in annotation, we require the corresponding summary to fulfill all attributes together. the data annotation pipeline is divided into four steps (figure 2 ). first, we carefully select the source texts from several widely used summarization datasets in news or dialogue domains. second, some automatic tools are leveraged to form a pool of candidate attributes as guidance for the next step. third, the annotators manually label five attributes to form a control attribute set and repeat the process multiple times. finally, the annotators write down the summary that meets each control attribute set. ## methods for setting baseline results on macsum, we propose three models following previous research on controllable text generation using prompt learning. with the same input and different prompts, the large pretrained model is able to generate different results for different tasks, such as summarization and translation @xcite @xcite . as shown in figure 4 , we leverage two types of prompt learning approaches to control the attributes of summaries, namely, hard prompt (hp) and soft prefix tuning (sp). we also test the combination of them, hp+sp. hard prompt (hp) uses the description of control attributes as the hard prompt. each attribute is for the hp model, the control attributes are prepended to the input to form a hard prompt. for the sp model, the selected prefix vectors are added together to form a control prefix. hp+sp contains both hard prompts and control prefixes. formed as ''attribute: value'', where ''attribute'' can be ''topic, speaker, length, extractiveness, specificity'', and ''value'' is the corresponding value (e.g., high or normal) of the attribute. we concatenate 5 control attributes using '';'' and prepend it to the input source. soft prefix (sp) follows li and liang (2021). we prepend external trainable parameters to both the encoder and decoder to control the summarization model. for controlling len, ext, and spe, we assign m prefix embeddings for each attribute value where m is a hyper-parameter meaning the length of prefix, i.e., prefix length. readers can refer to li and liang (2021) for implementation details. for example, for len: long, we assign e len:@xmath0:long , • • • , e m len:long ] where e j i is a vector with dimension of word embedding. and for controlling an input case with a set v of mixed requirements, we sum the embeddings of all control attribute values: @xmath1, • • • , @xmath2. and for controlling tpc and spk, we use the embeddings of input topics words e t pc and input speaker names e spk . this list of embedding vectors e is then prepended to each layer of the transformer-based summarization model as external key/value vectors in its self-attention operations. e t pc and e spk are prepended only to the input layer. hard prompt + soft prefix (hp+sp) combines both approaches by prepending the hard prompt of five attributes in hp and using prefix tuning in sp. ## experiments in this section, we present the implementation details, experimental results, and human evaluation of models on macsum dataset. ## analysis and discussion for a deeper understanding of the task of mixedattribute controllable summarization on mac-sum, we conduct analysis including attribute difficulty, attribute dependency, model pretraining, and present several example outputs for case studies. ## conclusion we propose macsum, a high-quality humanannotated benchmark for mixed-attribute controllable summarization. it contains 5 types of control attributes, including topic, speaker, length, extractiveness, and specificity. to the best of our knowledge, macsum is the first dataset with mixed attributes as well as human annotations. we explore the hard prompt and soft prefix models and evaluate them on macsum. results and analysis demonstrate that hard prompt models yield the best performance and also show this is a challenging task as a large gap between machine learning models and human still exists. future work can design more effective models for the mixed-attribute controllable summarization task, or explore mixed-attribute control on other generation tasks. case 1: topic defocus (mac-doc) attributes length: normal; extractiveness: full; specificity: normal; topic: education gold they quickly reopened the university of mosul, under a radically altered curriculum. some subjects would be banned -democracy and political thought, hotel management, tourism and archaeology. isis allows girls to go to school, in a segregated environment. hp the taliban, forbids all girls' education. but isis allows girls to go to school, albeit in a segregated environment. case 2: length against specificity (mac-doc) attributes length: short; extractiveness: normal; specificity: high; topic: campaign gold all variations of women feature in sunday's campaign video release by hillary clinton. hp clinton's new campaign website is focused on women and the progress of women in politics. hp+sp during her campaign, hillary clinton has focused on women's issues. her new campaign website is filled with women's pictures. case 3: extractiveness against readability (mac-dial) attributes length: normal; extractiveness: full; specificity: normal; topic: parallel marketing, angle; speaker: marketing gold marketing; either market it as the point of view; we could have parallel marketing s schemes; one where you've got one where it appeals to people that want to have the new device that looks cool, is fashionable; so um, i dunno we'll have to decide which which angle we're gonna go to or both; either market it together by getting control in a set colour or like you buy it with several; as a separate thing. hp+sp marketing, could have parallel marketing, schemes, one where it appeals to people that want to have the new device that looks cool; one that rather, than a kind of a need relationship with the device; people might not like, having a device, just looks nice; also a device, practically sound; decide which angle, gonna go to or both.