Unsupervised Style Transfer in News Headlines via Discrete Style Space (2023)

Abstract: “The goal of headline style transfer in this paper is to make a headline more attractive whilemaintaining its meaning. The absence of parallel training data is one of the main problems in thisfield. In this work, we design a discrete style space for unsupervised headline style transfer, shortfor D-HST. This model decomposes the style-dependent text generation into content-featureextraction and style modelling. Then, generation decoder receives input from content, style,and their mixing components. In particular, it is considered that textual style signal is moreabstract than the text itself. Therefore, we propose to model the style representation space asa discrete space, and each discrete point corresponds to a particular category of the styles thatcan be elicited by syntactic structure. Finally, we provide a new style-transfer dataset, namedas TechST, which focuses on transferring news headline into those that are more eye-catchingin technical social media. In the experiments, we develop two automatic evaluation metrics— style transfer rate (STR) and style-content trade-off (SCT) — along with a few traditionalcriteria to assess the overall effectiveness of the style transfer. In addition, the human evaluationis thoroughly conducted in terms of assessing the generation quality and creatively mimicking ascenario in which a user clicks on appealing headlines to determine the click-through rate. Ourresults indicate the D-HST achieves state-of-the-art results in these comprehensive evaluations. Introduction”

## introduction a style makes sense under pragmatic use and becomes a protocol to regularize the manner of communication @xcite . so, the task of text style transfer is to paraphrase the source text in a desired style-relevant application @xcite . in practical use, the style is data-driven and task-oriented in different area @xcite . the absence of parallel training data for a certain style is one of the difficult problems. continuous latent space mapping is a typical method for unsupervised style transfer to address the issue. @xcite model the latent space to a gaussian distribution. points in latent space are moved to the target representation with some style guidance. @xcite @xcite disentangle the continuous latent representation purely according to its content, and replace the source attribute to the target one. however, there are two problems of the continuous space approach. firstly, the style is highly abstract so that it is unstable and too sparse to accurately represent the style in the continuous space. second, the continuous vector-based representation is difficult to manipulate and cannot be examined at a finer level. to control the style transfer and enhance its explainability, several kinds of discrete signals are used to represent the style. for instance, @xcite @xcite employ mask-retrieve-generate strategy to decompose style attributes by wordlevel editing actions. but, these methods express styles in a highly discrete way which fail to capture the relationships between words or sentences. to more effectively describe the style in a highly abstract and discrete manner while also capturing the semantic relations in the texts, we propose a latent and discrete style space for headline style transfer, abbreviated as d-hst. this model decomposes style-dependent text generation into content-feature extraction and style modeling. therefore, we design a dual-encoder and a shared single-decoder framework to accomplish the overall generation. due to the lack of parallel training data, we have to synthesize adequate training pairs to accommodate the content extraction and the style modeling. given a target stylistic headline, we first automatically generate a content-similar input as well as style-consistent input for feeding the dual encoders. as the textual style signal is expected to be rather abstract and limited compared to the text itself, we propose to model the style representation space as a discrete space, with each discrete point denoting a particular category of the styles that can be elicited by syntactic structure. also, we provide a new style-transfer dataset derived from the real scenarios, named as techst, which transfers news headlines into the ones that are more attractive to readers. although several datasets are currently available for this purpose @xcite , but the appealing styles-such as humor and romance-are taken from fictional works of literature, which we believe makes them unsuitable for usage as an attractive style for headlines. in the experiments, we design two automatic evaluation metrics, including style transfer rate (str) and style-content trade-off (sct) -along with a few traditional criteria to assess the overall of the style transfer. additionally, the quality of the generation is thoroughly evaluated, and the click-through rate is calculated by creatively simulating a scenario in which a user clicks on attractive headlines. our findings show that the d-hst performs at the cutting edge in these thorough assessments. in conclusion, our article mainly has the following contributions. ## related work attractive headline generation it is crucial to generate eye-catching headlines for an article. @xcite proposes to generate attractive captions for images and videos with different styles. jin et al. (2020) introduces a parameter sharing scheme to generate eye-catchy headlines with three different styles, humorous, romantic, and clickbait. li et al. (2021) proposes a disentanglement-based model to generate attractive headlines for chinese news. we build upon this task by rewriting source headlines to attractive ones. text style transfer there are mainly three kinds of methods used in tst task. 1) modeling in the latent space mueller et al. (2017; liu et al. (2020) use continuous space revision to search for target space. shen et al. (2017; sun and zhu () learn a mapper function in source and target space. john et al. (2018; romanov et al. (2018; hu et al. (2017) explicitly disentangle content and style in latent space. however, the style is highly abstract so that it is unstable and too sparse to accurately represent the style in the continuous space. 2) prototype editing it is a word replacement method. li et al. (2018; tran et al. (2020) propose three-stage methods to replace stylist words with retrieved words in the target corpus. reid and zhong (2021) uses levenshtein editing to search target stylist words. these methods work well on content-preferences dataset, like sentiment, debias. 3) control code index keskar et al. (2019; dai et al. (2019) use a control sign embedding to controls the attribute of generated text. yi et al. (2021) to model the discrete style in an unsupervised fashion, we propose to inherit the third and fourth methods. specifically, we construct pseudo data to enrich the content-based parallel data and style-based para. further, different from the previously styled latent space, we model it as a discrete one based on the c c l 2 0 2 3 claim that style is highly abstract and more sparse compared to content. we will describe this in detail in the next section. ## methodology we are given samples @xmath0, y 2 , . . . y m } from the style dataset s. the objective of our task is to transfer a headline sentence to a new headline equipped with the style of the target data s, while maintaining its originally semantic content. ## tasks and datasets for the headline style transfer task, we focus on attractive news headline transfer on technology topics. technology news headlines are always formal. for example, "google document translation now generally available." is a common style for an event headline. on the contrary, technology blog headlines in social media tend to be special and catch readers' eyes. in this paper, we define this kind of headline as "attractive" style. to highlight the characteristics of style, the previous example can be transferred as "google goodness: document now available". the goal of this task is to transfer the formal news headlines to more attractive blog headlines in technology domain. ## experiments and results external knowledge as mentioned in section 3.3, external knowledge is used to estimate the style strength. to some extents, users' pageviews reflect attractiveness of the style. we first parsed all the syntactic structures of sentences in the style dataset. then, we calculate average-pageviews for each syntactic structure. the more average-pageviews the structure receives, the higher style score it has. we acknowledge that style isn't the only factor that affects pageviews, content also contributes to it. for example, headlines with syntactic structure like "np vp" are common, but some headlines with such structure may have high pageviews. to eliminate the impact of content, we add a pageview variance term. specifically, if sentences with same syntactic structure show little pageview variance, it is speculated that pageviews are determined by the syntactic structure. on the contrary, if the variance is significant, it suggests that other elements, such as content, are influencing pageviews. as such, the style score must be penalized. finally, we define our style score as: for gyafc dataset, no such corresponding information is provided, so we set all syntactic structures the same style score. experiment setup we use 6-layers transformers to train our model. each transformer has 8 attention heads and 768 dimensional hidden state. dropout with 0.1 was added to each layer in the transformer. encoder and decoder initialized from bart base. hyperparameters δ and ϵ in loss function are set to 0.5. in external knowledge building, we set ω = 2, µ = 0.05. we trained our model on a 3090 gpu for 20 epochs taking about 5 hours with gradient accumulation every 2 steps. we chose the best checkpoint for the testing through a validation process. baselines we compared the proposed model against the following three strong baseline approaches in text style transfer: bart+r @xcite is trained by fine-tune bart model with an extra bleu reward and style classification reward. this model uses parallel dataset. in order to meet our requirement of unsupervised style transfer setting, we used pseudo-parallel data x y cont and y as input and target in the following experiment. styins @xcite leverages the generative flow technique to extract stylistic properties from multiple instances to form a latent style space, and style representations are then sampled from this space. tsst @xcite proposes a retrieval-based context-aware style representation that uses an extra retriever module to alleviate the domain inconsistency in content and style. ## conclusion this paper presents an unsupervised model for headline style transfer. it consists of content, style and their mixing components, which are together fed to decoder for headline generation. in particular, we propose to extract the style features in a discrete style space, and each discrete point corresponds to a particular category of the styles. our system is comprehensively evaluated by both quantitative and qualitative metrics, and it produces cutting-edge outcomes in two typical datasets. our work can be applied in the scenarios of formality machine translation, politeness transfer in intelligent customer service, spoken language transfer in live broadcast delivery. it can also be followed by the task of paraphrase and data augmentation.