Surveying the FAIR ness of Annotation Tools: Difficult to find, difficult to reuse (2024)

Abstract: In the realm of Machine Learning and Deep Learning, there is a need for high-quality annotated data to train and evaluate supervised models. An extensive number of annotation tools have been developed to facilitate the data labelling process. However, finding the right tool is a demanding task involving thorough searching and testing. Hence, to effectively navigate the multitude of tools, it becomes essential to ensure their findability, accessibility, interoperability, and reusability (FAIR). This survey addresses the FAIRness of existing annotation software by evaluating 50 different tools against the FAIR principles for research software (FAIR4RS). The study indicates that while being accessible and interoperable, annotation tools are difficult to find and reuse. In addition, there is a need to establish community standards for annotation software development, documentation, and distribution.

## introduction coinciding with the rise of machine learning (ml), annotation projects started being conducted to create labelled datasets for the training and testing of models, especially those based on supervised approaches @xcite . a wide range of software has been developed to address data labelling. existing annotation tools vary in terms of supported modality (i. e., text, image, video, audio), access type (e. g., desktop, webbased, etc.), license @xcite , and annotation task(s) they are designed for @xcite . this sheer abundance of different annotation tools hinders the search, access, and choice of an appropriate tool. researchers often spend a lot of time downloading and installing tools that turn out to be irrelevant to their projects @xcite . to improve the findability, accessibility, interoperability, and reusability (fair) of research artifacts such as annotation tools, the fair @xcite and fair for research software @xcite principles have been proposed. although previous surveys of annotation tools exist @xcite @xcite @xcite , none of them evaluated their fairness. however, the development of fair annotation tools is essential to facilitate knowledge discovery and to ensure transparent research. this paper addresses this gap by assessing a range of 50 different annotation tools according to the fair4rs principles. our contributions can be summarised as follows: the rest of the paper is structured as follows: section 2 discusses the annotation lifecycle, fair and fair4rs principles in more detail. section 3 describes the annotation tools selection process and introduces our interpretation of the fair4rs principles. section 4 and section 5 present the evaluation results and main findings, respectively. section 6 discusses the limitations of our study. concluding remarks are provided in section 7. ## methods in this study, we performed a manual assessment of annotation tools against the fair4rs principles. we refrained from using automatic solutions primarily due to the variability in results mentioned in section 2.2. furthermore, howfairis is the only tool based on fair4rs, designed specifically for analysing the compliance of github/gitlab repositories with the principles. consequently, its applicability is limited to tools hosted on those platforms. due to the huge amount of existing tools @xcite and time constraints, we limited the evaluation to 50 annotation tools. as a first step, we randomly selected annotation tools surveyed by @xcite . however, those are specifically developed for text data annotation. to make the set of tools more diverse in terms of covered modalities, we conducted a search on google scholar to find publications related to annotation and corpus creation that mention or cite annotation software. in addition, we looked for tools on platforms such as european language grid (elg, rehm, 2023), zenodo foot_6 , sourceforge foot_7 and software heritage foot_8 . we did not consider tools that were archived or have become part of another project (e. g., webanno, eckart de castilho et al., 2016), were not found (e. g., a publication exists but the link to the home page or source code does not work) or are for crowd-sourcing purposes (amazon mechanical turk foot_9 ). since the fair4rs principles do not serve as a set of strict rules but rather as a guideline, they are not rigidly defined, sometimes allowing for a broad range of interpretations. therefore, below we introduce our interpretations and the evaluation strategies defined and followed in this study foot_10 . f1. software is assigned a globally unique and persistent identifier. an annotation tool should have a globally unique and persistent identifier (pid), such as doi, which assures longevity and consistently points to the software despite changes in its location, content or other attributes. thus, we investigated whether a tool is available on platforms that provide pids, i. e., software heritage and zenodo. other widely utilised software publishing services, such as github, gitlab, or sourceforge, are not suitable as the urls they offer cannot be considered persistent. f1.1. components of the software representing levels of granularity are assigned distinct identifiers. in addition to the annotation tool itself, distinct pids should be assigned to all its components. thus, following the software granularity levels schema offered by chue hong et al. (2022), we researched whether files, directories, commits, releases, and other tool attributes possess pids. in contrast to f1, we considered only software heritage since it assigns distinct pids to every digital object component compared to zenodo, which provides individual dois only for various versions. f1.2. different versions of the software are assigned distinct identifiers. each release of an annotation tool should be assigned a distinct pid allowing users to track its development and refer to a specific version they utilised. similar to f1, we checked the presence of tools on both software heritage and zenodo. f2. software is described with rich metadata. metadata should be semantically structured, i. e., being both human and machine-readable. it should contain a thorough description of an annotation tool allowing users to understand how to utilise and replicate it without looking into its source code. metadata is considered to be rich when it goes beyond basic information. to define the minimum metadata we followed the bioschemas com-putationaltool foot_11 . bioschemas is an effort to improve findability in life sciences by relying on the widely used schema.org foot_12 vocabulary. although bioschemas is domain-specific, it includes general types and properties to describe research artifacts such as datasets and software. unlike other vocabularies, bioschemas offers minimum, recommended, and optional property types, making it easier to define rich metadata. accordingly, the minimum metadata of an annotation tool should include name, url, and description. the metadata is considered to be rich if at least one additional property from any marginality level is provided foot_13 . f3. metadata clearly and explicitly include the identifier of the software they describe. if an annotation tool is assigned a pid, it should be referenced by it in the respective structured metadata. thus, in case f1 is not fulfilled, f3 fails as well. f4. metadata are fair, searchable and indexable. metadata is fair when it is semantically structured. therefore, if an annotation tool fails f2 due to the lack of structured metadata, it automatically fails f4. any metadata exposed via web pages in a format understood by search engines or deposited in a repository/registry with search functionality is indexable. a1. software is retrievable by its identifier using a standardised communications protocol. an annotation tool should be accessed through a commonly used communication protocol such as hypertext transfer protocol (http), hypertext transfer protocol secure (https) or file transfer protocol (ftp). ## results we present an assessment of 50 annotation tools based on the 17 fair4rs principles. the complete list of tools and the results are provided in according to our results, none of the tools fully adhere to all of the 17 principles. the maximum number of fulfilled principles across the tools is 13, while the minimum is 3. out of 50 tools, 6 have reached the maximum of 13 and only one fulfills 3 principles. on average, tools comply with approximately 9 principles. when it comes to findability, our analysis reveals that 29 annotation tools satisfy the requirements for f1, while the rest fall short as they are hosted on fulfill both i1 and i2. in particular, 40 annotation tools support standard data formats for both input and output. only 10 tools partially adhere to i1, most of which (9 tools) do not provide details on either input or output formats, and one application (pundit) writes data into notebooks that are not considered standard. all but one annotation software (lighttag) fulfill i2 by offering qualified reference to other objects in either structured json/xml metadata files or in unstructured documentation using standard naming conventions. commonly referenced objects are programming languages, compatible browsers, and datasets. assessing reusability, we note that none of the tools fully comply with r1 as they fail f2 or r1.2. there are 11 tools that achieve partial fulfillment. the majority of tools (43) provide a clear description of the license in either semantically structured or unstructured metadata, thereby adhering to r1.1. however, only 12 tools offer license information in both metadata types. most annotation tools (31) specify license in unstructured readme/license files or on the project webpage. doccano, textinator, and gate teamware do not update license information in a structured metadata file. in terms of provenance, only via fully complies with r1.2. there are 11 tools that partially adhere to the principle since they provide citation information and details on origin, development history, and/or contributors/authors. the remaining software either have minimal provenance, limited to release history or authors/contributors or lack it entirely. similar to the i2 principle, only lightag fails to satisfy the criteria for r2. finally, when it comes to r3, results indicate that only doctag and gate teamware fully fulfil it. among the rest of the tools, 21 have partial compliance and 27 fail the principle. ## discussion the fairness assessment indicates that while annotation tools are accessible and interoperable, there is a strong need for improving their findability and reusability. in particular, a large number of tools lack pids on various levels (files, releases, etc.). even in cases when an annotation software possesses a pid, there is no reference to it in the respective metadata. these factors hinder the accurate and unambiguous citation of a tool as well as tracking its developmental changes over time. the results also show that the vast majority of annotation software suffers from the absence of semantically structured metadata. as was noted in section 4, even if there is one, some details such as license or input/output data formats tend to be provided in unstructured formats. furthermore, such tools do not have structured metadata published separately either. consequently, tools are less discoverable and linkable to other related systems. it becomes difficult to reuse and replicate these annotation applications without delving into the implementation details or testing them. the situation is even worse when it comes to provenance descriptions. most tools do not have fine-grained documentation of their origin and development history. the lack of sufficient provenance information contributes to low reproducibility and makes it difficult to build upon existing annotation tools. additionally, there is a clear need for agreedupon community standards and best practices regarding annotation tools functionalities, metadata vocabulary and import/export file formats. the absence of those influenced the assessment results. ## limitations while our analysis is rather comprehensive, it is not without limitations. first, the manual approach to evaluation is error prone. it would be beneficial to align our findings with the results from automatic fairness assessment solutions, namely using howfairis (at least for tools hosted on github/gitlab). second, tools were not tested with respect to their executability. thus, there is no guarantee that all surveyed tools can actually be installed and run properly. this also means that a tool could possibly have a specific feature but it is not stated in its metadata. third, when it comes to annotation tools, not only the software itself should be fair but the labelled data it produces should be fair, too. however, the fairness evaluation of annotated data produced by or with annotation tools is out of scope with regard to this study and we leave it for future work. finally, as previously noted, the fair4rs principles are aspirational in nature. thus, the interpretations defined in this paper should be treated as initial suggestions rather than rigid definitions. ## conclusion in this paper, we investigated how annotation tools comply with the fair4rs principles. we performed a manual evaluation of 50 tools following interpretations of the fair4rs principles adapted specifically to annotation software. the findings reveal that the findability and reusability of annotation tools require improvement. specifically, the lack of pids, semantically structured metadata and detailed provenance are the most problematic aspects. additionally, the study shows that there is a demand for agreed-upon community standards for annotation software management.