E dit E val: An Instruction-Based Benchmark for Text Improvements (2024)

Abstract: Evaluation of text generation to date has primarily focused on content created sequentially, rather than improvements on a piece of text. Writing, however, is naturally an iterative and incremental process that requires expertise in different modular skills such as fixing outdated information or making the writing style more consistent. Even so, comprehensive evaluation of a model’s capacity to perform these skills and the ability to edit remains sparse. This work introduces EditEval: An instruction-based, benchmark and evaluation suite that leverages high-quality existing and new datasets in English for the automatic evaluation of editing capabilities, such as making text more cohesive and paraphrasing. We evaluate several pre-trained models, which shows that InstructGPT and PEER on average perform the best, but that most baselines fall below the supervised state-of-the-art, particularly when neutralizing and updating information. Our analysis also shows that commonly used metrics for editing tasks do not always correlate well, and that prompts leading to the strongest performance do not necessarily elicit strong performance across different models. Through the release of this benchmark (code and data available at https://github.com/facebookresearch/EditEval) and a publicly available leaderboard challenge, we hope to unlock future work on developing models more capable of controllable and iterative editing.

## introduction large pre-trained language models have shown impressive text generation capabilities for a wide variety of tasks such as question answering, textual entailment, and summarization @xcite @xcite @xcite . however, to date, most work employing language models has focused on generating immutable text in a single pass. this is in stark contrast to the way in which humans develop articles of text, which is naturally an iterative process of small steps, each with a precise purpose @xcite . this is a crucial process because it allows for analysis of "what's working, what isn't, and what it still needs" and adaptation to these needs along the way @xcite . in many cases, a needed change may only become apparent after much of the text is created, such as in the case of a reorganization or fixing inconsistencies or contradictions @xcite . in this way, the current paradigm of generating text passages in a single pass can be severely limiting. additionally, the current paradigm of continuous left-to-right generation is less controllable and not flexible to human-in-the-loop collaboration and feedback, and this absence of experienced human mediation in the writing process can be highly detrimental to the quality of the final product @xcite . while there are some existing production tools geared towards working with humans to compose articles and emails, such as grammarly 3 , smart compose from google 4 and text predictions from microsoft 5 , a majority focus on sentence completion rather than iteratively improving upon prior text. a more powerful editing barack obama and hillary clinton were both running for the democratic party nominationcandidate nominees of the democratic party for the 2008 presidential election. on june 3rd, obama received enough endorsements to rightly clinch the nomination. obama went on to win the general election against republican john mccain and became the current44th president of the united states of america. clinton went on to serve as the 67th united states secretary of state, serving within the obama administration. in the 2016 presidential election, she became the nominee of the democratic party, becoming the first woman to be a nominee of a major u.s. political party. clinton, however, loselost the general election to donald trump. clinton's third memoir, what happened, features an account of her loss in the 2016 election. ## related work several multitask evaluation benchmarks have been open-sourced to the community to support progress in natural language understanding including glue @xcite , superglue @xcite , decanlp @xcite , and gem @xcite . these datasets, however, focus on a broad set of tasks in nlp (e.g., question answering, reading comprehension, and textual entailment). while all of these tasks are critical to natural language understanding, editeval focuses on curating a benchmark for measuring a model's capability to improve and edit text. there are several datasets which focus on iterative text revisions in the domain of wikipedia @xcite , academic essays @xcite , and news articles @xcite . these works, however, focus on one particular domain and in some cases, a particular style like argumentative writing @xcite . editeval, on the other hand, includes examples from multiple domains: wikipedia, wikinews, news articles, and arxiv. iterater @xcite is perhaps closet to editeval in that it provides iterative tasks from multiple domains, but it has a limited number of such tasks: fluency, coherence, clarity, style, and meaning-changed. because this is a great starting point, we have included iterater in editeval, and we additionally develop prompts for these tasks since iterater is not instruction-based. moreover, unlike iterater, editeval includes novel datasets for tasks such as updating text using new information and neutralizing the text, which are core components of editing a factually-correct and unbiased article. ## the editeval benchmark editeval is an instruction-based benchmark for iterative text generation/modification. editeval sources existing high-quality datasets-most with human annotations-containing tasks relevant to editing. these datasets are combined into a unified evaluation tool and can be evaluated with any metric provided in editeval. a task here refers to a type of edit (e.g., simplification), and the specific task dictates which set of prompts to be used (e.g., simplify this text), the full set of which is enumerated in appendix b. we consider seven editing tasks in editeval. the corresponding datasets for each task included in editeval are enumerated in the input text is the original text before revision, and the gold edits are the target edits for that specific task type. lastly, the reference documents provide textual information from external articles or documents that are relevant to the task. the task that requires reference documents is updating, and otherwise, the reference documents field is empty. the datasets in editeval were selected if they test a capability relevant to the art of editing and contain human-annotated gold edits, if possible. we also endeavored to include datasets that are broadly used by the community. the datasets in editeval are by no means exhaustive, but the editeval framework is flexible such that it can easily extend to new datasets and metrics in future versions. ## metrics the metrics we included in editeval are ones that are (1) shown to have significant correlation with human judgement for a task in editeval and (2) commonly used to benchmark one of the datasets in editeval. below, we discuss some of the main metrics. appendix c describes these and additional metrics in greater detail. ## baselines for each baseline, we use greedy decoding, and we do not perform any task-specific fine-tuning or in-context learning. we evaluate on editeval using the following baselines: we evaluate the text-davinci-001 version described in @xcite since, at the time of writing, details about the training process for text-davinci-002 were not publicly available. ## formatting we evaluate these baselines on their general capability to accomplish each task when prompted in natural language in a zero-shot fashion. because there are a diverse set of ways in which to instruct for each task, we manually construct a set of 3-11 prompts in order to more robustly evaluate performance. for each task prompt t and input i, the model is given a formatted input following the template: task: t\ninput: i\noutput: with an additional field for references, should they be required. figure 2 shows an example of an input including references. for tasks without references, we exclude this field. some slight modification to this template were made. for example, tk-instruct expects the prompt to be prefixed by the string "definition:" rather than "task:"). for preprocessing, we used the natural language toolkit (nltk) package @xcite for tokenizing the text. update information obama went on to win the general election against republican john mccain and became the current president of the united states. [0] barack obama | barack hussein obama ii is an american politician who served as the 44th president of the united states. obama went on to win the general election against republican john mccain and became the 44th president of the united states task: input: reference: output: figure 2 : example of inputs formatted when evaluating the baseline models. each input is evaluated with a set of prompts that are determined by the task type. ## results we summarize results in instructgpt and peer perform the best overall. in when using averages across prompts and using the minimum, instructgpt performs the best, but peer performs the best when using the maximum across prompts. most baselines lag substantially behind the supervised sota, especially in the task of updating and neutralization. we show the supervised state-of-the-art results in the final row of the most challenging tasks do not necessarily have the highest variance across models. in observing figure 3a , we see that the tasks which have the largest variance across models (assessed using the interquartile range or iqr) are fluency and updating information. this is despite the fact that the fluency datasets are arguably easier (i.e., many of the models come close to the supervised scores) than the updating datasets, exemplifying that difficulty and robustness can be independent axes. jf-leg also appears to be easier than itr-f (average sari of 45.1 versus 38.2), which is understandable since jfleg sources from the toefl exam (primarily simpler and conversational sentences), while iterater sources technical sentences from wikipedia, arxiv, and wikinews. likewise, turk-corpus seems on average to be slightly easier than asset, which is expected since it includes more diverse simplifications than turkcorpus. peer has the highest total variance, but opt and gpt-3 are less robust to different prompts. from figure 3 , we observe that the peer models have the largest variance in performance overall (as measured by the larger iqr). if we compute the standard deviation across prompts and normalize by the mean (cv in optimizing prompts according to maximum performance and according to robustness to different models can be orthogonal objectives. ideally, we would like to create prompts that achieve the highest performance using the best baseline, but also perform reliably well for any model. in assessing variance from figure 4 , we see that certain prompts stand out as less robust to different models relative to others. for example, for neutralization, prompts #1, 2, and 7 are less robust likely because they use uncommon language such as "remove points of views" or "neutralize this text". some of the prompts which are less robust for simplification (prompts #4, 7) and paraphrasing (prompts #4, 6) are sometimes ones that are less specific such as "rewrite this text" versus "rewrite this with different wording"-in the case of the former, an empirical assessment shows that the models seem to more often copy the original text and make fewer modifications. unfortunately, choosing prompts that achieve the maximum score does not always entail prompts which are the most robust-prompt #5 for clarity achieves the maximum but has the largest variance in performance or iqr. some of the tasks exhibit a great degree of outlier behavior (coherence, paraphrasing, or neutralization), which is either due to t0 performing exceedingly low or instructgpt/peer performing exceedingly well. other tasks such as fluency and updating seem to have prompts with a similar range of variance. commonly used metrics are not always wellcorrelated. we measure the pearson correlation between each pair of metrics using evaluation scores for all baselines, which is shown in figure 5 , and find that many of the commonly used metrics do not always correlate well with each other, a finding echoed by prior works @xcite , which focuses on the task of grammatical error correction. we exclude peer in this analysis since it shows exceedingly strong performance in some cases, and we exclude the updating datasets since they are of a very different nature from the other datasets. we find that while families of variants like bleu and ibleu as well as rouge and updaterouge show strong correlation within each respective set (> 0.97), the two sets are inversely correlated with one another (-0.29 to -0.1). rouge actually appears to be the metric that most conflicts with all other metrics, whereas gleu seems to be the metric that is most in harmony with the rest (0.41-0.76). though sari is not correlated with rouge, it is the metric which shows the strongest correlation with em-diff (0.83) and updaterouge (0.7). ## discussion we present editeval, a benchmark composed of handcrafted, task-specific instructions for several editing datasets across multiple domains. edite-val is a means of evaluating models for these tasks according to multiple popular metrics, all within a single, unified tool. we show that while models such as instructgpt have impressive performance, in general the baselines lag behind the supervised state-of-the-art, particularly for the task of updating and neutralization. our analysis of metrics and prompts shows that several popular metrics are not well-correlated, even conflicting at times, and that small changes in the wording of a prompt can lead to substantial changes in performance and robustness to different models. this suggests further c2 ), are prompts #6 and 11 (fluency), 4 (clarity), 2 (coherence), 8 and 10 (simplification), 3 (paraphrasing), 2 (neutralization) and 2 and 1 (updating). certain prompts evoke more variation across models due to factors such as using less frequently used language or being too unspecific. work is needed to develop models comprehensively capable of executing editing tasks in addition to developing a standardized way of measuring editing capabilities and systematically selecting prompts. in releasing this work, we hope to bolster work in which language models are utilized for text generation that is iterative, more controllable, collaborative, and capable of revising and correcting text.