A Review of Human Evaluation for Style Transfer (2021)

Abstract: This paper reviews and summarizes human evaluation practices described in 97 style transfer papers with respect to three main evaluation aspects: style transfer, meaning preservation, and fluency. In principle, evaluations by human raters should be the most reliable. However, in style transfer papers, we find that protocols for human evaluations are often underspecified and not standardized, which hampers the reproducibility of research in this field and progress toward better human and automatic evaluation methods.

## introduction style transfer (st) in nlp refers to a broad spectrum of text generation tasks that aim to rewrite a sentence to change a specific attribute of language use in context while preserving others (e.g., make an informal request formal, despite the growing interest that st receives from the nlp community, progress is hampered by the lack of standardized evaluation practices. one practical aspect that contributes to this problem is the conceptualization and formalization of styles in natural language. according to a survey of neural style transfer by @xcite , in the context of nlp, st is used to refer to tasks where styles follow a linguistically motivated dimension of language variation (e.g., formality), and also to tasks where the distinction between style and content is implicitly defined by data (e.g., positive or negative sentiment). across these tasks, st quality is usually evaluated across three dimensions: style transfer (has the desired attributed been changed as intended?), meaning preservation (are the other attributes preserved?), and fluency (is the output well-formed?) @xcite . given the large spectrum of stylistic attributes studied and the lack of naturally occurring references for the associated st tasks, prior work emphasizes the limitations of automatic evaluation. as a result, progress in this growing field relies heavily on human evaluations to quantify progress among the three evaluation aspects. inspired by recent critiques of human evaluations of natural language generation (nlg) systems @xcite @xcite @xcite , we conduct a structured review of human evaluation for neural style transfer systems as their evaluation is primarily based on human judgments. concretely, out of the 97 papers we reviewed, 69 of them resort to human evaluation (figure 1 ), where it is treated either as a substitute for automatic metrics or as a more reliable evaluation. this paper summarizes the findings of the review and raises the following concerns on current human evaluation practices: ## reviewing st human evaluation paper selection we select papers for this study from the list compiled by @xcite who conduct a comprehensive review of st that covers the task formulation; evaluation metrics; opinion papers and deep-learning based textual st methods. the paper list contains more than 100 papers and is publicly available ( https://github.com/ fuzhenxin/style-transfer-in-text ). we reviewed all papers in this list to determine whether they conduct either human or automatic evaluation on system outputs for st, and therefore should be included in our structured review. we did not review papers for text simplification, as it has been studied separately @xcite and metrics for automatic evaluation have been widely adopted @xcite . our final list consists of 97 papers: 86 of them are from top-tier nlp and ai venues: acl, eacl, emnlp, naacl, tacl, ieee, aaai, neurips, icml, and iclr, and the remaining 11 are pre-prints which have not been peer-reviewed. review structure we review each paper based on a predefined set of criteria ( global criteria can be split into three categories which describe: (1) the st stylistic attribute, ( 2 ) four details about the annotators and their compensation, and (3) four general design choices of the human evaluation that are not tied to a specific evaluation dimension. for the dimension-specific criteria we repurpose the following operationalisation attributes introduced by @xcite : form of response elicitation (direct vs. relative), details on type of collected responses, size/scale of rating instrument, and statistics computed on response values. finally, we also collect information on the quality criterion for each dimension (i.e., the wording used in the paper to refer to the specific evaluation dimension). process the review was conducted by the authors of this survey. we first went through each of the 97 papers and highlighted the sections which included mentions of human evaluation. next, we developed our criteria by creating a draft based on prior work and issues we had observed in the first step. we then discussed and refined the criteria after testing it on a subset of the papers. once the criteria were finalized, we split the papers evenly between all the authors. annotations were spotchecked to resolve uncertainties or concerns that were found in reviewing dimension-specific criteria (e.g., scale of rating instrument is not explicitly global criteria task(s) st task(s) covered presence of human annotation presence of human evaluation annotators' details details on annotator's background/recruitment process annotators' compensation annotator's payment for annotating each instance quality control quality control methods followed to ensure reliability of collected judgments annotations' availability availability of collected judgments evaluated systems number of different systems present in human evaluation size of evaluated instance set number of instances evaluated for each system size of annotation set per instance number of collected annotations for each annotated instance agreement statistics presence of inter-annotator agreement statistics sampling method method for selecting instances for evaluation from the original test sets dimension-specific criteria presence of human evaluation whether there exists human evaluation for a specific aspect quality criterion name quality criterion of evaluated attribute as mentioned in the paper direct response elicitation presence of direct assessment (i.e., each instance is evaluated on its own right) relative judgment type (if applicable) type of relative judgment (e.g., pairwise, ranking, best) direct rating scale (if applicable) list of possible response values presence of lineage reference whether the evaluation reuses an evaluation framework from prior work lineage source (if applicable) citation of prior evaluation framework ## findings based on our review, we first discuss trends of stylistic attributes as discussed in st research through the years ( §3.1), followed by global criteria of human evaluation ( §3.2), and then turn to domain-specific criteria ( §3.3).