Integrating INCE p TION into larger annotation processes (2024)

Abstract: Annotation tools are increasingly only steps in a larger process into which they need to be integrated, for instance by calling out to web services for labeling support or importing documents from external sources. This requires certain capabilities that annotation tools need to support in order to keep up. Here, we define the respective requirements and how popular annotation tools support them. As a demonstration for how these can be implemented, we adapted INCEpTION, a semantic annotation platform offering intelligent assistance and knowledge management. For instance, support for a range of APIs has been added to INCEpTION through which it can be controlled and which allow it to interact with external services such as authorization services, crowdsourcing platforms, terminology services or machine learning services. Additionally, we introduce new capabilities that allow custom rendering of XML documents and even the ability to add new JavaScript-based editor plugins, thereby making INCEpTION usable in an even wider range of annotation tasks.

## introduction/motivation annotated data is crucial for many branches of science and industry. it is used in supervised learning to train and evaluate machine learning models @xcite and has been the catalyst as well as limiting factor for the deep learning revolution @xcite . large language models often require high-quality annotated data, be it for (instruction) fine-tuning or their evaluation @xcite @xcite . as the processes producing and consuming annotations become more complex, annotation tools need to be able to act as a part in these larger processes. for instance, they need to be embedded in a crowdsourcing pipeline @xcite , integrate external knowledge bases @xcite , or provide functionality to call machine learning models for annotation support @xcite . they also need to be customizable in order to cope with the ever-demanding change in requirements. if the functionality for a task is not implemented yet, they need to be extensible so that these new features can be easily retrofitted. we survey the required capabilities in five areas relevant to customization and integration into larger processes: annotator management, task design, process integration, machine learning services and external knowledge and discuss if and how popular annotation tools support them. in addition, we describe how inception has implemented them to serve as a role model for future implementations. ## related work over the years, many annotation tools have been developed that target different use cases and come with different capabilities. we discuss some older but popular as well as some more recently published text annotation tools (see @xcite for a more comprehensive overview). most tools considered are free open source tools. label studio and potato are freemium tools that require a paid license for certain functionalities or use-cases. prodigy is commercial software. design choices tend to be made based on whether an annotation tool is mainly instanceoriented or document-oriented. many recent tools are instance-oriented and focusing on high throughput (e.g., prodigy @xcite , la-bel studio, gate teamware 2 @xcite , alanno @xcite , label sleuth @xcite , or potato @xcite ). they try to get annotators to label as many instances as possible in the shortest amount of time. often, these tools only support labelling the entire instance. some support span and relation annotation tasks, but tend to focus on very alanno doccano gate label sleuth label studio potato prodigy brat medtator webanno inception annotator management am-1: multi-user ---am-2: workload mgmt. dyn stat dyn n/a dyn* dyn n/a stat n/a stat dyn am-3: reclaim abandoned ---------am-4: self-sign-up l idp l n/a l, idp* url n/a -n/a -url, idp task design td-1: customizable ui tagset tagset templ. tagset templ. templ. templ. schema schema schema schema td-2: document layout ---------process integration pi-1: api -r r r r -l --r r pi-2: event notifications --------ml services ml-1: ml support bi p p bi p, r bi p, l p, r p p, bi p, bi, r ml-2: active learning --* ---knowledge bases kb-1: rdf/sparql ----------kb-2: generic lookup --------- ## requirements and contributions in the following subsections, we identify several requirements that annotation tools should meet in order to integrate well into a larger process and how inception meets these requirements. these are noted as e.g. am-1 always. ## conclusion we have discussed recent developments in the the free and open source annotation tool inception which allows it to be integrated as a step into larger processes and which allow it to be customized using format and editor plugins so the tool can be used with a wider range of document types and for a wider range of annotation tasks. we have compared the tool to the state-of-the-art and see that based on the capabilities discussed here, incep-tion is one of the most versatile tools in its peer group. that said, we see further opportunities for innovative annotation user interfaces (e.g. to better accommodate annotation tasks related to large language models) as well as in for supporting a wider range of document types.