FALTE : A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022)

Abstract: A growing swath of NLP research is tackling problems related to generating long text, including tasks such as open-ended story generation, summarization, dialogue, and more. However, we currently lack appropriate tools to evaluate these long outputs of generation models: classic automatic metrics such as ROUGE have been shown to perform poorly, and newer learned metrics do not necessarily work wellfor all tasks and domains of text. Human rating and error analysis remains a crucial component for any evaluation of long text generation. In this paper, we introduce FALTE, a web-based annotation toolkit designed to address this shortcoming. Our tool allows researchers to collect fine-grained judgments of text quality from crowdworkers using an error taxonomy specific to the downstream task. Using the taskinterface, annotators can select and assign error labels to text span selections in an incremental paragraph-level annotation workflow. The latter functionality is designed to simplify the document-level task into smaller units and reduce cognitive load on the annotators. Our tool has previously been used to run a large-scale annotation study that evaluates the coherence of long generated summaries, demonstrating its utility.

## introduction recent years have seen a significant improvement in the generation capabilities of large language models @xcite @xcite , across tasks such as machine translation, open-ended story generation, summarization, and others. as these models generate extremely fluent and human-like text, their errors are more subtle than those of prior models and harder to detect @xcite . overlap-based automatic metrics such as rouge @xcite , bleu @xcite , and bertscore @xcite have historically been the most popular ## falte tool description falte is a web-based annotation toolkit designed to help researchers run large-scale annotation stud-ies evaluating the quality of model generated text. the dataset collected through falte has the following form: annotators highlight errors @xmath0. annotators incrementally proceed through the document and annotate as many errors as they can identify in the text. in this section, we will outline the workflow for the two stakeholders: 1) task designers (researchers) that employ the tool to run these studies, and 2) crowd annotators that interact with our web interface and provide annotations for the quality of long text. the overall workflow is outlined in figure 2 . ## related work reference-based evaluation is the most popular evaluation paradigm for generation models. these include overlap-based metrics @xcite @xcite , or distributional similarity metrics @xcite , and others. however, recent work has shown that these do not correlate with human judgments of quality @xcite @xcite . human evaluation of generation quality is generally considered to be more reliable, although there do not exist any fixed protocols for conducting these studies @xcite . in recent work, both likert scale rating and a/b testing based evaluation frameworks have been widely used @xcite . however, across both these frameworks, tasks are generally designed to elicit document-level quality judgments from crowdworkers that are insufficient to measure the quality of generated text @xcite @xcite . particularly, @xcite show that crowd annotators often conflate multiple dimensions of quality, and tend to primarily focus on surface properties like grammaticality while evaluating summaries. therefore, in our task design, we focus on fine-grained error annotations that allow annotators to clearly distinguish between the different error categories and their occurrences. the document-level task design of the prior work discussed above is quite straightforward to set up using the basic ui components provided by crowdsourcing platforms such as mechanical turk. however, creating a user-friendly interface for finegrained annotation collection is much more challenging. @xcite create a task interface for fine-grained annotations of short generated text. in contrast, our iterative tool design is motivated by prior crowdsourcing research @xcite that shows that worker performance and attention increases with an incremental task design for longer tasks. moreover, decomposition into smaller paragraph-level annotations also reduces the cognitive load on the annotator @xcite . ## conclusion we present falte, a web-based annotation tool to collect fine-grained error annotations for text. it provides an easy-to-use interface to annotate and submit fine-grained annotations and is equipped with capabilities such as navigational flexibility and coreference highlighting that are specifically designed for better user experience while annotating long documents. on the task designer side, our tool is highly customizable: task designers can define their own error taxonomy, error category specifications, and annotation granularity. therefore, it can accommodate a wide variety of evaluation objectives, e.g. different dimensions of quality like coherence or factuality, language or task-specific taxonomies, and more. we hope that falte can support the design and launch of fine-grained human evaluation studies in the future.