Don’t Take This Out of Context!: On the Need for Contextual Models and Evaluations for Stylistic Rewriting (2023)

Abstract: Most existing stylistic text rewriting methods and evaluation metrics operate on a sentence level, but ignoring the broader context of the text can lead to preferring generic, ambiguous, and incoherent rewrites. In this paper, we investigate integrating the preceding textual context into both the \textit{rewriting} and \textit{evaluation} stages of stylistic text rewriting, and introduce a new composite contextual evaluation metric \texttt{CtxSimFit} that combines similarity to the original sentence with contextual cohesiveness. We comparatively evaluate non-contextual and contextual rewrites in formality, toxicity, and sentiment transfer tasks. Our experiments show that humans significantly prefer contextual rewrites as more fitting and natural over non-contextual ones, yet existing sentence-level automatic metrics (e.g., ROUGE, SBERT) correlate poorly with human preferences ( \rho =0–0.3). In contrast, human preferences are much better reflected by both our novel \texttt{CtxSimFit} ( \rho =0.7–0.9) as well as proposed context-infused versions of common metrics ( \rho =0.4–0.7). Overall, our findings highlight the importance of integrating context into the generation and especially the evaluation stages of stylistic text rewriting.

## introduction existing methods for stylistic text rewriting, i.e., adapting the text to a particular style while preserving its originally intended meaning, often fail to account for a statement's context (e.g., @xcite @xcite @xcite @xcite . as a result, these systems may change the speakers' original communicative intents and generate contextually irrelevant and generic outputs. for example, in figure 1 , a non-contextual model rewriting an informal response to a formal one simply replaces words with more formal synonyms, whereas a contextual rewriting model can use the broader conversational context to produce a more specific and natural formal rewrite. informal response: "i know, right? i'm drowning in them." preceding dialog: "i can't believe how much work we have to do." ## background & related work in this section, we discuss the increasing interest in incorporating context into nlp tasks and motivate the significance of context during the rephrasing and evaluation phases of stylistic text rewriting. stylistic text rewriting despite being introduced over ten years ago @xcite , current methods for stylistic rewriting (e.g. @xcite @xcite @xcite @xcite @xcite still rely solely on parallel source-to-target sentence pairs, primarily due to a lack of datasets that include contextual information. while new models have emerged that do not require parallel data for training @xcite @xcite , they also operate without contextual information. building on some preliminary research that explored context in small customtrained seq2seq rewriting models @xcite and large language models for exemplar-based conversation-level rewriting @xcite , we extend the investigation to large language models with defined style attributes like formality, toxicity, and sentiment. importantly, we also explore the need for context in evaluations in addition to modeling, and propose a new suite of contextualized metrics for automatic evaluation. ## task and datasets to measure the importance of context in rewriting, we scope our investigations around three specific attribute controls: formality, sentiment, and toxicity, chosen because they necessitate varying degrees of meaning preservation and style intensity. we present statistics for each of the datasets used in our rewriting tasks in ## modeling context in rewriting in this section, we introduce our methodology for contextual stylistic rewriting utilizing large language models (llms) and in-context learning. we conduct a comparison of three types of rewrites: those generated with context, those generated without context, and those generated with a random context (as a counterfactual baseline). figure 2 provides a visual representation of our approach. ## contextual human evaluation since in realistic rewriting scenarios, context will always be available and crucial to users who wish to rewrite their dialogue utterances or story sentences @xcite , we start by conducting a contextual human evaluation to gauge user preferences between non-contextual and contextual rewrites. this contextual human evaluation is a departure from most previous work which has predominantly not used context ( §2). ## non-contextual automatic evaluation overall, our contextual human evaluations reveal a general preference for contextual rewrites over non-contextual ones. given that prior work primarily evaluated utterance-level rewrites in both human and automatic evaluations, it raises the question of how well non-contextual automatic metrics mirror human preferences. in this section, we investigate commonly used metrics in previous work @xcite for meaning preservation, style strength, and fluency. ## contextual automatic evaluation as shown in the previous section, non-contextual automatic metrics, especially for meaning preservation, are not sufficient to evaluate the performance of rewriting models. to address this, incorporating context into the evaluation process is necessary for better representing realistic downstream use cases. drawing inspiration from reference-free metrics in dialog evaluation @xcite , which considers both the dialog context and generated responses to assess responses within the dialogue history, we propose including context into existing automatic evaluation metrics and further introduce ctxsimfit, a new contextual metric.