On the interaction of automatic evaluation and task framing in headline style transfer (2020)

Abstract: An ongoing debate in the NLG community concerns the best way to evaluate systems, with human evaluation often being considered the most reliable method, compared to corpus-based metrics. However, tasks involving subtle textual differences, such as style transfer, tend to be hard for humans to perform. In this paper, we propose an evaluation method for this task based on purposely-trained classifiers, showing that it better reflects system differences than traditional metrics such as BLEU.

## introduction and background the evaluation of natural language generation (nlg) systems is intrinsically complex. this is in part due to the virtually open-ended range of possible ways of expressing content, making it difficult to determine a 'gold standard' or 'ground truth'. as a result, there has been growing scepticism in the field surrounding the validity of corpus-based metrics, primarily because of their weak or highly variable correlations with human judgments @xcite @xcite . human evaluation is generally viewed as the most desirable method to assess generated text @xcite . in their recent comprehensive survey on the evaluation of nlg systems, @xcite stress that it is important that any used untrained automatic measure (such as bleu, rouge, meteor, etc) correlates well with human judgements. at the same time, human evaluation also presents its challenges and there have been calls for the development of new, more reliable metrics @xcite . beyond the costs associated with using humans in the loop during development, it also appears that certain linguistic judgment tasks are hard for humans to perform reliably. for instance, human judges show relatively low agreement in the presence of syntactic variation @xcite . by the same token, @xcite observe at best moderate correlations between human raters on stylistic dimensions such as politeness, colloquialism and naturalness. closer to the concerns of the present work, it has recently been shown that humans find it difficult to identify subtle stylistic differences between texts. de mattei et al. (2020b) presented three independent judges with headlines from two italian newspapers with distinct ideological leanings and in-house editorial styles. when asked to classify the headlines according to which newspaper they thought they came from, all three annotators performed the task with low accuracy (ranging from 57% to 62%). furthermore, agreement was very low (krippendorff's α = 0.16). agreement was similarly low on classifying automatically generated headlines (α = 0.13 or 0.14 for two different generation settings). these results suggest that human evaluation is not viable, or at least not sufficient, for this task. in this work we focus on the same style-transfer task using headlines from newspapers in italian, but address the question of whether a series of classifiers that monitor both style strength as well as content preservation, the core aspects of style transfer @xcite @xcite , can shed light on differences between models. we also add some untrained automatic metrics for evaluation. as observed above, the fact that humans cannot perform this task reliably makes it impossible to choose such metrics based on good correlations with human judgement @xcite . therefore, relying on previous work, we compare the insights gained from our classifiers with those obtained from bleu @xcite and rouge @xcite , since they are commonly used metrics to assess performance for content preservation and summarisation. other common metrics such as meteor @xcite and bleurt @xcite , which in principle would be desirable to use, are not applicable to our use case as they require resources not available for italian. more specifically, we train a classifier which, given a headline coming from one of two newspapers with distinct ideological leanings and in-house styles, can identify the provenance of the headline with high accuracy. we use this (the 'main' classifier) to evaluate the success of a model in regenerating a headline from one newspaper, in the style of the other. we add two further consistency checks, both of which aim at content assessment, and are carried out using additional classifiers trained for the purpose: (a) a model's output headline should still be compatible in content with the original headline; (b) the output headline should also be compatible in content with the article to which it pertains. a headline is deemed to be (re)generated successfully in a different style if both (a) and (b) are satisfied, and the main classifier's decision as to its provenance should be reversed, relative to its decision on the original headline. a core element in our setup is testing our evaluation classifiers/strategies in different scenarios that arise from different ways of framing the style transfer task, and different degrees of data availability. indeed, we frame the task either as a translation problem, where a headline is rewritten in the target style or as a summarisation problem, where the target headline is generated starting from the source article, using a summarisation model trained on target style. the two settings differ in their needs in terms of training data as well as in their ability to perform the two core aspects of style transfer (style strength and content preservation). we observe how evaluation is affected by the different settings, and how this should be taken into account when deciding what the best model is. the data and task settings also lend themselves well as material for a shared task, and they have indeed been used, with the summarisation system described here as baseline, in the context of the evalita 2020 campaign for italian nlp @xcite . collection we used a corpus of around 152k article-headline pairs from two wide circulation italian newspapers at opposite ends of the political spectrum namely la repubblica (left-wing) and il giornale (right-wing) provided by de @xcite . the data is balanced across the two sources. though we are concerned with headlines, full articles are used in two ways: (a) alignment; and (b) the consistency check classifiers (see section 4 for details). for the former, we leverage the alignment procedure proposed by @xcite and we split our dataset into strongly aligned, weakly aligned and non-aligned news. the purpose of alignment is to control for potential topic biases in the two newspapers so as to better disentangle newspaper-specific style. additionally, this information is useful in the creation of our datasets, specifically as it addresses the need for parallel data for our evaluation classifiers and the translation-based model (see below). alignment we compute the tf-idf vectors of all the articles of both newspapers and create subsets of relevant news filtering by date, i.e. considering only news which were published approximately within the same, short time interval for the two sources. on the tf-idf vectors we then compute cosine similarities for all news in the resulting subset, rank them, and retain only the alignments that are we choose two different thresholds: one is stricter (> 0.5) and we use it to select the best alignments; the other one is looser (> 0.185, @xmath0.5). ## evaluation our fully automatic strategy is based on a series of classifiers to assess style strength and content preservation. for style, we train a single classifier (main). for content, we train two classifiers that perform two 'consitency checks': one ensures that the two headlines (original and transformed) are still compatible (hh classifier); the other ensures that the headline is still compatible with the original article (ah classifier). see also figure 1a . in what follows we describe these classifiers in more detail. when discussing results, we will show how the contribution of each classifier is crucial towards a comprehensive evaluation. main classifier the main classifier uses a pretrained bert encoder with a linear classifier on top fine-tuned with a batch size of 256 and sequences truncated at 32 tokens for 6 epochs with learning rate 1e-05. given a headline, this classifier can distinguish the two sources with an f-score of approximately 80% (see ## results and discussion with better-aligned data (bottom panel), the picture is more nuanced. here, the main comparison is between two systems trained on strongly aligned data, one of which (s2s2) has additional, weakly aligned data. the overall compliancy score suggests that this improves style transfer (and this system is also the top performing one over all, also outperforming s2s1 and sum). as for content preservation (ah and hh scores), s2s3 is marginally better on average for hh, but not for ah, where the two systems are tied. overall, the results of the classification-based evaluation also highlight a difference between a summarisation-based system (sum), which tends to be better at content preservation, compared to a translation-based style transfer setup (especially s2s2) which transfers style better. clearly, a corpus-based metric such as bleu fails to capture these distinctions, but here does not appear informative even just for assessing content preservation. one aspect that will require further investigation, since we do not have a clear explanation for it as of now, is the performance difference between the two translation directions. indeed, transforming a la repubblica headline into a il giornale headline appears more difficult than transforming headlines in the opposite directions, under most settings. ## conclusions this paper addressed the issue of how to evaluate style transfer. we explicitly compared systems in terms of the extent to which they preserve content, and their success at transferring style. the latter is known to be hard for humans to evaluate @xcite . our aim was primarily to see to what extent different evaluation strategies based on purposely trained classifiers could distinguish between models, insofar as they perform better at either of these tasks and in different training scenarios. our findings suggest that our proposed combination of classifiers focused on both content and style transfer can potentially help to distinguish models in terms of their strengths. interestingly, a commonly used metric such as bleu does not seem to be informative in our experiments, not even for the content preservation aspects. to the extent that stylistic distinctions remain hard for humans to evaluate in setups such as the one used here, a classification-based approach with consistency checks for content preservation is a promising way forward, especially to support development in a relatively cheap and effective way. future work will have to determine how the various metrics we have used relate to each other (especially our classifiers and bleu/rouge), and whether human judgement can be successfully brought back, and in case in what form, at some stage of the evaluation process.