EEVEE : An Easy Annotation Tool for Natural Language Processing (2024)

Abstract: Annotation tools are the starting point for creating Natural Language Processing (NLP) datasets. There is a wide variety of tools available; setting up these tools is however a hindrance. We propose Eevee, an annotation tool focused on simplicity, efficiency, and ease of use. It can run directly in the browser (no setup required) and uses tab-separated files (as opposed to character offsets or task-specific formats) for annotation. It allows for annotation of multiple tasks on a single dataset and supports four task-types: sequence labeling, span labeling, text classification and seq2seq.

## introduction annotated datasets are of paramount importance to the natural language processing (nlp) community. their use is at the core of research, e.g. for training models, evaluating models, and analyzing trends. one of the first considerations when creating an annotated dataset is which annotation tool to choose. there is a variety of (open-source) tools readily available with extensive feature-sets. we were motivated by the following observed difficulties with existing tools when designing eevee: to obtain annotations on the token level, an often cumbersome conversion is necessary. based on these observations, we propose eevee: a simple, free, and flexible annotation tool built around tab-separated files. it is written in javascript and runs directly in the browser. it can also be saved as a desktop application and run offline. the intuitive interface allows novice users to import a dataset and set up multiple annotation tasks quickly. the graphical user interface has two main pages: the setup page (section 2) and the annotation page (section 3). it supports tab-separated files and raw text input (section 4.1). we perform a case study on ner annotation with the system usability scale from usability engineering (section 6). finally, we compare eevee to other toolkits (section 7). ## setup page figure 1 illustrates the setup page where the user can define the annotation environment. tasks can be configured in the task field (figure 1 , top right), allowing the user to specify the input column (for the input text) and output column (for the target task), as well as adding the desired labels. labels can also be imported automatically from the annotated file (if it already contains annotations), and a default label can be set for empty annotations. for utterance-level tasks (i.e. classification), the annotation is stored in a comment above the text, in the form "# @xmath0. to facilitate reproducibility and improve the ease of setup, the tool allows the import and export of all settings to configuration files that users can create for predefined tasks (more details in section 4.1). once a dataset has been imported, the tabular data field (figure 1 , bottom right) offers a simple overview of the raw data belonging to each utterance. the user can add new columns or remove existing ones to achieve the desired result. this makes eevee an easy-to-use tool for extending or editing tab-separated data as well (see section 4.1). once the data and tasks are ready, the user simply clicks "annotate" (figure 1 , bottom left) to continue to the "annotation page" (see section 3). ## annotation page figure 2 illustrates an example of a ner task in the annotation interface. the user is presented with a clean, minimal annotation environment. the annotation process has been designed with efficiency in mind, enabling the user to navigate the interface also through keyboard shortcuts. the navigation bar (figure 2 , bottom right) enables navigation between utterances and, similar to prodigy @xcite , setting the status of a given task for a given utterance. the status can be set to four values: completed, wrong, unsure, and cleared (i.e. none). this overall status is reflected in the progress bar (figure 2 , top right), allowing the user to spot missing and unsure annotations easily. the progress bar is also useful when continuing annotation on a previously saved annotation file. eevee provides two different annotation modes for label-based tasks: the keyboard mode and the search mode. with the keyboard mode (figure 2 ), the user can use the number keys to select labels and click/select the part of the input where the label should apply (for utterance-level tasks, simply pressing the number key is sufficient). in search mode, a small pop-up appears after selecting a word or span (see figure 3 ), allowing the user to find the desired label quickly. if there are more than ten labels, eevee defaults to search mode. finally, the annotation file can be exported (figure 2 , bottom left). the current datetime can be appended to distinguish between different export versions. ## tasks in this section, we will describe the annotation data format used by eevee (for import and export, importing text files is also supported), and we will discuss all the supported task types as well as the configuration files for the setups. ## compatability with other services a recent development is the huggingface datasets library @xcite , which has indexed 62k+ datasets in two years. this library does not share the text directly but through a python api. we provide a convenient python script that automatically downloads data from the datasets library and converts it to the tab-separated format of eevee. one of the toolkits that operates on tab-separated formats is machamp @xcite , which is focused on multi-task learning. machamp supports all the tasks that are included in eevee. for convenience, we provide a conversion script that takes eevee files as input and outputs a machamp configuration file and the corresponding training command. ## system usability study 6.1 procedure to assess the usability of eevee, we conduct a case study with two annotators on two tasks, named entity annotation (span labeling), and german dialect identification (classification). before annotating with eevee, annotators spent four months labeling named entities (ne) directly on tab-separated text files in a text editor using bio encoding and dialect identification (did) labels as utterance-level metadata. in this case study, we ask both annotators to conduct the same ne and did annotation tasks on a set of new documents, similar to previous ones but using the newly introduced eevee. during eevee training, we present a 12-minute tutorial video explaining the setup and annotation pages to the annotators and provide them with tabseparated unannotated files and the json configuration files. two annotators separately annotate the same eight documents, four from wikipedia (wiki) and four from twitter (x, tweet), summing up to 14.2k tokens and 16 working hours per person. foot_2 ## comparison to other annotation toolkits we compare eevee to other available toolkits in other techniques for improving annotation speed need more tuning and setup and could lead to biases. for example, active learning could lead to model bias @xcite and coloring relevant words for a task @xcite could lead to biases towards these indicators. we leave the user management up to the organizer of the annotation efforts and prioritize the simplicity in tool setup. furthermore, since eevee does not need installation, it does not store or send any data to the network, which is beneficial for data privacy. ## conclusion we introduce eevee, an annotation toolkit focused on easy setup and usability. it runs directly in the browser and allows for annotation of multiple tasks. in addition, it provides convenience scripts for usage with other libraries. eevee's main distinguishing features, in contrast to other toolkits, are the simplicity of its setup and use, as well as annotation directly on the token level (tab-separated files). to evaluate the tool, we conducted a case study using the system usability scale, resulting in high usability scores. we also qualitatively surveyed the annotators' experience and noted that they would prefer to use the tool again for annotation.