POTATO : The Portable Text Annotation Tool (2022)

Abstract: We present POTATO, the Portable text annotation tool, a free, fully open-sourced annotation system that 1) supports labeling many types of text and multimodal data; 2) offers easy-to-configure features to maximize the productivity of both deployers and annotators (convenient templates for common ML/NLP tasks, active learning, keypress shortcuts, keyword highlights, tooltips); and 3) supports a high degree of customization (editable UI, inserting pre-screening questions, attention and qualification tests). Experiments over two annotation tasks suggest that POTATO improves labeling speed through its specially-designed productivity features, especially for long documents and complex tasks. POTATO is available at https://github.com/davidjurgens/potato and will continue to be updated.

## introduction much of nlp requires annotated data. as nlp has tried to tackle increasingly more complex linguistic phenomena or diverse labeling and classification tasks, the annotation process has increased in complexity-yet the need for and benefit of large labeled datasets remain @xcite . modern annotation tools like label studio @xcite , light-tag @xcite , doccano @xcite , and prodigy (explosion, 2017) have partially filled this gap, providing a variety of solutions to different types of annotations. however, these tools each bring their own challenges: requiring external access, limiting visual configurability for complex tasks, or even costing hundreds of dollars-prohibitive for small groups. we introduce potato, the portable text annotation tool, which allows practitioners to quickly design and deploy complex annotations tasks. potato has been designed, developed, and tested over a two-year period with the following design goals in mind (figure 1 ): 1) high accessibility. potato is open-sourced under the mit license and free to anyone. potato is built with minimal dependencies to allow researchers and developers to easily build and integrate their own features. 2) easy to deploy. potato comes with templates covering a wide range of annotation tasks like bestworst-scaling, text classification, and multi-modal conversation. anyone can start a new annotation project with simple configurations. potato is also rapidly and easily deployable in local and webbased configurations and has seamless integration with common crowdsourcing platforms like amazon mechanical turk and prolific. potato flexibly supports diverse annotation needs. with our specially designed schema rendering and custom rendering mechanism, potato allows nearly all kinds of text annotation tasks and is visually customizable to support complex task designs and layout. ## architecture and design potato is written in python and focuses on portability and simplicity in annotation and deployment. the user interface is created through an extensible html template and configuration file, which allows practitioners to quickly develop and deploy common setups like likert-scale annotation while also supporting extensive display customization. the potato server populates the interface with data provided by the operator and supports displaying any html-supported modality, e.g., text or images. an overview of the architecture is shown in figure 2 . data management potato loads data in common file formats, such as delimited files or newlinedelimited json. this allows it to ingest data in the json format supplied by the twitter or reddit apis, as well as other types of data used by the deployer. all formats are converted into internal data structures that link the deployer-selected instance id to annotations. at a minimum, deployers must specify which field represents the unique instance id and, for most tasks, the text to be annotated. the data may contain other columns which will be included in the final output and can optionally be used in customized visualizations. annotation schema rendering potato allows deployers to select one or more forms of annotation for their data using predefined schema types, such as multiple choice or best-worst scaling. deployers fill out which options should be shown and then each scheme is rendered into html upon the completion of loading the data. annotation instructions can be provided as an external url that annotators may view or using html text shown in potato that annotators may collapse vertically to free up screen space. potato provides default html templates that automatically lay out each scheme's annotation questions. however, deployers may additionally customize the html templates and select their own layout using jinja expressions (e.g., {{text}}; jinja) to specify where parts of the annotation task and data are populated within the user-defined template. user management annotators create accounts and then log in to view their tasks using a secure user management system. when used with crowdsourcing platforms, potato also allows workers to directly jump to the annotation task using their crowdsourcing user id. for each new annotator, potato automatically assigns instances as configured by the deployer and all the annotations are recorded on the backend. when logging out and back in, annotators resume at the most recent unannotated item. potato also allows deployers to, with minimum configurations, set up pre-and post-screening questions (e.g., having annotators provide demographics or complete psychological questionnaires), pre-study tests, and attention tests to identify unreliable annotators. ## comparison with existing systems potato has been developed to fill a key niche left by existing systems for providing visual customization, easy annotator-support features, and rapid development. the ultimate goal is to provide simple and comprehensive solutions to common annotation tasks as well as allow personalized design for complex tasks. productivity potato is designed to maximize the productivity of both annotators and deployers. while most of the existing annotation tools generally focus on labeling data, potato supports a series of features that can help with the entire data annotation pipeline. potato allows easilycustomizable keyboard shortcuts to allow efficient annotation. for visually or cognitively challenging settings, potato allows conditional highlights, which helps to reduce task complexity and focus annotators' attention. finally, active learning can reduce the annotation time needed to curate an informative dataset. often, annotation tools that offer a highly customizable annotation interface do not also implement productivity features: the open-sourced version of labelstudio @xcite only supports keypress shortcuts, while flat @xcite supports none of these features. for deployers, potato allows seamless integration with common crowdsourcing platforms like amazon mechanical turk and prolific. quality control collective high-quality and reliable annotations is the ultimate goal of data labeling tasks and is usually the key to the success of the final ml/nlp systems. potato comes with a series of quality control feature which helps deployers to reliably collect annotations. while some other annotation systems like label studio and we-banno also support agreement calculation, none of the existing systems come with features that help deployers to improve the annotation quality and analyze factors affecting it. potato allows deployers to easily set up prestudy qualification tests (annotators have to pass a small test to participate in the full annotation) and attention tests (attention test questions are randomly inserted in the annotation queue as configured by the deployer) to identify unreliable annotators before, within, and after annotation. potato also allows deployers to freely insert survey questions before and after the annotation phase. deployers can easily define different pages of pre-and post-screening questions with minimum effort and potato also provides a series of templates for common survey questions like user consent and demographic information. recent studies suggest that the background of annotators has substantial effects on the quality and bias of data labeling and further affects the fairness of ml/nlp models (e.g., @xcite . with potato, researchers can easily collect background information of annotators and analyze the effect of annotator backgrounds on data labeling. accessibility potato is free to use and actively maintained. while commercial annotation tools like prodigy (explosion, 2017) can come with more functionality, these tools are expensive; for example, prodigy costs @xmath059 usd per person per month. these costs are potentially prohibitive for students and researchers without access. potato is fully open-sourced and is deployed with minimum dependencies. moreover, in addition to giving the flexibility to freely define uis and annotation settings, potato allows researchers to easily share their annotation settings with a simple yaml file, aiding in replication and future extension of prior work. ## experimental analysis potato was designed to minimize set-up time and per-instance annotation time while maintaining annotation accuracy. therefore, we conduct a user study to compare the time for setup and annotation time per instance compared to its free competitors. we compare potato's performance on two long and complex tasks, each involving identifying themes and causes in narrative summaries of reports of completed suicides: annotation setup one annotator completed 50 annotations for task 1 and 100 annotations for task 2 on potato and a number of freely available and feature-rich annotation tools: microsoft excel, doccano @xcite , label studio @xcite , and lighttag @xcite . the annotator was highly familiar with the task and classification scheme, having annotated 1000+ instances of each task prior to this user study, so familiarity with the codebook was not a factor. given the complexity of the task, potato was initialized with 118 keywords to associate with conditional highlights (e.g., retir*, layoff, work*), key bindings, and classes included tooltips summarizing each label. to test the effect of these productivity-enhancing features, we include a version of potato that does not include these features, called inconvenient potato. to ensure the same level of familiarity with each document, the documents annotated with each tool are randomly sampled from a larger set of 203,531 documents. for each annotation tool we measure the time to set up an annotation task without counting time taken to (1) install and familiarize ourselves with the tool (e.g., trial and error in set-up), ( 2 ) generate the annotation data files, and (3) write the properties of each label and keywords. each tool was configured as comparably as possible (e.g., keypress shortcuts were always enabled and active learning disabled). to reduce the influence of initial unfamiliarity with each tool on per-instance timing, the annotator completed 10 untimed instances. then, for each tool, we record the time spent annotating per instance. results across both annotation tasks, potato is approximately 30-50% faster than competitors like excel, label studio, and lighttag (figure 4bc ). annotating short documents is comparable in time to doccano (4b), while long documents are slightly faster in potato (4c). without convenience features like conditional highlighting, key mappings, and tooltips, potato's per-instance annotation time increases to be more comparable with other tools. we conclude that the difference in perinstance annotation time is likely attributable to these design features. including convenience features increases task setup time (figure 4a ), taking just over 4 extra minutes to configure at set-up. our results suggest that, compared to using potato with convenience features, the base setup without convenience features has lower overall task time (including setup) until an annotator has seen ∼20 long documents or ∼100 short documents. we note that potato without convenience features takes less time to set up than label studio and lighttag; with these three features, potato takes roughly the same amount of time to set up as doccano, even though doccano only supports one feature (keypress shortcut). the two tasks we chose share features with many common nlp annotation tasks that make them well-suited to a system like potato. we highlight two comparative observations across the interface from the user study. first, doccano and potato have the most annotator-friendly interfaces, which allow for fast coding. for instance, custom keypress shortcuts allow us to create 22 different bindings that make logical sense to the annotator. unlike other tools, these two tools did not require any use of the mouse (e.g., others required pressing submit with the mouse), which reduced the annotation time; in short document tasks, where the time to read the document is low, these time savings become especially important. the default page layout in potato also better supports content interpretation; for example, the text and labels frequently fit on one page with no scrolling, and by placing the text on top of the labels, the annotator did not need to scroll down in order to read the text-and since the annotator had the keypress bindings memorized, the task could often be accomplished with no scrolling. finally, since other tools often required uploading data to external servers, there was often a load time of 1-2 seconds per document; again, saving this time with locally-deployed tools was especially salient with short documents. second, both tasks involve assigning a large number of independent labels. the keyword highlights allow the annotator to quickly identify which subset of these labels are likely to be relevant to the document, while the key mappings allow them to quick apply the correct labels. keyword highlights are particularly useful for longer documents (e.g., task 1), because they help identify the text that is likely relevant to a given label, which is often buried in a large amount of irrelevant text (e.g., most labels apply to one of ∼13 sentences). additionally, if none of the labels applied to a document, the annotator needed to ensure that she did not overlook a relevant phrase in the long text; in potato the lack of relevant keywords allowed us to quickly confirm that none of the labels applied, while without keyword highlights, the annotator read these documents twice. the time savings associated with keyword highlights likely explains the slight per-instance advantage potato has over doccano for long documents but not short documents. ## conclusion and future plans potato distinguishes itself with a comprehensive suite of productivity-enhancing features that allow annotators to efficiently and accurately label data and researchers to quickly configure complex tasks on a wide range of data types. potato was created both for the computational scholar and the overburdened student or crowdworker, looking to annotate more data in their limited time. potato has been in continuous development for over two years and will continue to be developed to support new task designs, easier management, and faster annotation. for management, we aim to have (1) a unified gui for deployers to create new tasks and manage existing tasks, (2) a gui that supports real-time monitoring of annotation process, mirroring tools like webanno @xcite , and (3) integration with common social media platforms to display content with original interfaces (e.g. displaying tweets with their native ui). for annotators, we aim to support simple linguistic search to help annotators find and prioritize instances to annotate, and to support personalization in aspects such as annotators' visualization and keybindings. we also plan to conduct experiments and explore different design choices to reduce annotator burn-out.