S tyle BART : Decorate Pretrained Model with Style Adapters for Unsupervised Stylistic Headline Generation (2023)

Abstract: Stylistic headline generation is the task to generate a headline that not only summarizes the content of an article, but also reflects a desired style that attracts users. As style-specific article-headline pairs are scarce, previous researches focus on unsupervised approaches with a standard headline generation dataset and mono-style corpora. In this work, we follow this line and propose StyleBART, an unsupervised approach for stylistic headline generation. Our method decorates the pretrained BART model with adapters that are responsible for different styles and allows the generation of headlines with diverse styles by simply switching the adapters. Different from previous works, StyleBART separates the task of style learning and headline generation, making it possible to freely combine the base model and the style adapters during inference. We further propose an inverse paraphrasing task to enhance the style adapters. Extensive automatic and human evaluations show that StyleBART achieves new state-of-the-art performance in the unsupervised stylistic headline generation task, producing high-quality headlines with the desired style.

## introduction the sequence-to-sequence-based neural headline generation (hg) model @xcite @xcite has demonstrated its ability to generate factual, concise, and fluent headlines @xcite . yet, the headlines are also expected to have stylistic attributes to draw more attention of the audiences. to address the problem, @xcite propose the task of stylistic headline generation (shg), which aims to generate a headline with a specified style such as humorous, romantic and clickbaity. however, acquiring enough parallel data for shg is almost impossible, as the creation of headlines with specific styles demands creativity and often consumes significant effort. hence, researchers @xcite in turn explore unsupervised approaches which only require a standard headline generation dataset and non-parallel stylistic text corpora. some existing works propose a pipeline method @xcite @xcite which firstly generates a plain headline and then introduces the pre-specified style with a style transfer model. however, this pipeline method require two models during inference, which brings additional latency and storage cost. some other works @xcite jointly learn a plain headline generation model on news-headline pairs, and a denoising autoencoder on the stylistic text corpus. however, these approaches require to carefully design the scheduling and parameter sharing mechanisms between tasks. moreover, as the task of headline generation and style learning are entangled, the entire model has to be rebuilt when facing a new style or task. in this paper, we propose stylebart, an unsupervised shg model based on the bart model @xcite . instead of using a multitask learning framework, stylebart disentangles the style and the specific task (e.g. headline generation) by the design of model architecture and training strategy. intuitively, this enables style-bart to separately train the style and task modules, then combine them as required during inference (figure 1 ). hence, stylebart is more flexible and training-efficient compared with baselines @xcite which require training models for all style and task combinations. specifically, stylebart utilizes style adapters as plug-and-play modules for the style control. the style adapters are learned at the pretraining stage through the inverse paraphrasing task on the style corpus, while the base hg model is learned at the fine-tuning stage on news-headline pairs. during inference, we achieve unsupervised shg by switching to the target style adapter. in this way, style-bart has the same high decoding efficiency as @xcite , while overcoming its problems in task scheduling and inefficient training. we evaluate stylebart using the same three shg tasks described in @xcite . both automatic and human evaluations demonstrate the superiority of our method over baselines. ## conclusion we propose stylebart, an unsupervised stylistic generation method, which enables to freely combine the downstream tasks and styles by disentangling style learning and downstream task learning. experimental results show that our model can generate content-relevant and style-intensive headlines, and can be extended to other stylistic generation tasks.