Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation (2023)

Abstract: Fine-grained, span-level human evaluation has emerged as a reliable and robust method for evaluating text generation tasks such as summarization, simplification, machine translation and news generation, and the derived annotations have been useful for training automatic metrics and improving language models. However, existing annotation tools implemented for these evaluation frameworks lack the adaptability to be extended to different domains or languages, or modify annotation settings according to user needs; and, the absence of a unified annotated data format inhibits the research in multi-task learning. In this paper, we introduce Thresh, a unified, customizable and deployable platform for fine-grained evaluation. With a single YAML configuration file, users can build and test an annotation interface for any framework within minutes – all in one web browser window. To facilitate collaboration and sharing, Thresh provides a community hub that hosts a collection of fine-grained frameworks and corresponding annotations made and collected by the community, covering a wide range of NLP tasks. For deployment, Thresh offers multiple options for any scale of annotation projects from small manual inspections to large crowdsourcing ones. Additionally, we introduce a Python library to streamline the entire process from typology design and deployment to annotation processing. Thresh is publicly accessible at https://thresh.tools.

## introduction as modern large language models are able to generate human-level quality text @xcite , the evaluation of these models becomes increasingly challenging. recent work has shown traditional surface-level evaluation methods such as pairwise comparison or likert-scale ratings become less reliable @xcite due to the close performance of these llms. to address this, several fine-grained human figure 1 : examples of fine-grained evaluation frameworks implemented on thresh. in order: salsa @xcite , mqm @xcite , scarecrow @xcite . however, each of these evaluation frameworks releases its own dedicated annotation interface that is difficult to modify or adapt to different evaluation schemes, thus limiting the customizablility. for example, scarecrow's typology @xcite , which is designed for open-ended text generation framework task released evaluation mqm @xcite translation ✓ frank @xcite summarization ✓ snac @xcite narrative summarization ✓ scarecrow @xcite open-ended generation ✓ salsa @xcite simplification ✓ errant @xcite grammar error correction ✗ fg-rlhf @xcite fine-grained rlhf ✓ inspection multipit @xcite paraphrase generation ✗ cwzcc zamboanga chavacano ✗ (himoro and pareja-lora, 2020) spell checking propaganda propaganda analysis ✓ @xcite arxivedits @xcite scientific text revision ✓ for news, may require modifications when applied to other domains such as story or scientific writing. frameworks like mqm @xcite only allow selections of the spans in the target sentence, restricting the ability to select the associated source spans in error categories such as mistranslation. furthermore, modern llms are ideally evaluated on multiple tasks @xcite , but the lack of a unified annotation tool makes this process inconvenient. considering the recent success of multi-task instruction fine-tuning @xcite , a standardized annotation format would enable research in multi-task learning with fine-grained human feedback. to this end, we present thresh : a unified and customizable platform for building, distributing and orchestrating fine-grained human evaluation for text generation in an efficient and easy-to-use manner. our platform allows users to create, test and deploy an evaluation framework within minutes, all in a single browser window and has already been used to orchestrate large-scale data annotation @xcite . thresh also serves as a community hub for fine-grained evaluation frameworks and annotation data, all presented in a unified format. figure 1 displays three examples of evaluation frameworks built on thresh. the following are the design principles of thresh: annotation tool. accessible and replicable annotation tools have been a persistent goal for nlp tasks. @xcite introduces brat, the first web browser-based annotation tool and yimam et al. ( 2013 ) further improves brat on speed and label configuration. in recent years, a new generation of universal annotation tools have been introduced by academia and industry, including prodigy @xcite , doccano @xcite , lighttag @xcite , and potato @xcite . focusing on universality, these tools allow authors to add custom ui elements such as multiple choice questions, text boxes or pairwise comparison. however, these surface-level annotation options are not sufficient to implement complex typology setups demanded by fine-grained evaluation, which are typically structured by decision trees @xcite . thresh addresses this gap by recursively building the interface, which allows for nested questions. besides, thresh encourages sharing and reproducibility by providing a community hub where users can upload their new or use existing fine-grained frameworks and annotated data. ## fine-grained text evaluation thresh formulates fine-grained text evaluation as two components: span selection and span annotation. during development, users define their annotation typology and interface features using a yaml template (see sec 4 and fig 5 for more details). based on the configuration, thresh then constructs an annotation interface that integrates both components, as illustrated in figures 2 and 3 . ## interactive interface builder to alleviate the time consuming process of customizing and hosting front-end code -even building custom databases in some cases -thresh implements an in-browser interface builder, which allows users to create, test and deploy a fine-grained interface within a single web browser page, as depicted in figure 5 . users write a yaml template to construct their interface and provide data with a json file. the compile button allows users to preview their interface, and the deploy button presents instructions for different deployment options, which are described in §5. template hub. as thresh aims to facilitate easy use and distribution of fine-grained evaluation frameworks, it provides a template hub that makes it simple for any nlp practitioner to access a framework with their own data. alongside the 10 tutorial templates that explain each interface feature, the annotation builder currently includes 11 widely used inspection and evaluation typologies across major text generation tasks. to upload a framework to thresh, users can create a github pull request with their typology's yaml file, which is merged publicly. we also include other features to facilitate sharing and replication. users can add a citation flag along with a bibtex citation, which creates a cite this typology button in the annotation builder, a paper_link flag, which adds a link to their research paper in the builder and on deployment, and a demo_data_link flag which creates a view demo data button to allow viewers to use the interface with example data. for testing, users can paste data into the interface builder interactively, and for deployment can link to data files. data can be blank or come with existing annotations, in which case the annotations will be appropriately parsed, verified and rendered. unified data model. as shown in ## deployment managing and collecting fine-grained annotations becomes bulky at scale, we thus release supplementary tools to deploy interfaces quickly or programmatically, and integrate loading annotations directly into python. this includes the thresh library 5 , which is useful for compiling interfaces and loading annotations. we support the following deployment types as shown in figure 5 : in addition, we release demo code for users to host their interface on their own domain without cloning the thresh repository. additionally, we support lightweight database integration (such as with google firebase foot_4 ) for all deployment types, allowing users to connect their own database to any annotation setup. ## conclusion we present thresh , a unified, customizable, and deployable platform for fine-grained text evaluation. thresh offers extensive customization via a simple yaml configuration file, and facilitates a community hub for sharing frameworks and annotations. the platform also ensures seamless deployment for any scale of annotation projects and introduces a python library to further ease the process from typology design to annotation processing.