Unleashing annotations with T ext A nnotator: Multimedia, multi-perspective document views for ubiquitous annotation (2021)

Abstract: We argue that mainly due to technical innovation in the landscape of annotation tools, a conceptual change in annotation models and processes is also on the horizon. It is diagnosed that these changes are bound up with multi-media and multi-perspective facilities of annotation tools, in particular when considering virtual reality (VR) and augmented reality (AR) applications, their potential ubiquitous use, and the exploitation of externally trained natural language pre-processing methods. Such developments potentially lead to a dynamic and exploratory heuristic construction of the annotation process. With TextAnnotator an annotation suite is introduced which focuses on multi-mediality and multi-perspectivity with an interoperable set of task-specific annotation modules (e.g., for word classification, rhetorical structures, dependency trees, semantic roles, and more) and their linkage to VR and mobile implementations. The basic architecture and usage of TextAnnotator is described and related to the above mentioned shifts in the field.

## motivation annotation in and for computational linguistics @xcite underwent technical and conceptual developments from xml-based annotation formats to integrated gate @xcite or uima @xcite frameworks @xcite . one reason for that development is that annotation (regardless of the annotated media such as texts, images, music, video, and so on) is bound to annotation tools, usually one annotation tool per annotation task or purpose @xcite ). annotation tools are themselves subject to (technical) development (see, for instance, the annotation of rhetorical relations @xcite ). furthermore, annotation is often part of a machine learning (ml) pipeline where machine learned applications are trained on annotated data @xcite , so that they can later perform annotations automatically on larger data sets. this is most explicitly expressed in the matter/mama annotation model @xcite . in order to secure interoperability and data exchange in this dynamic landscape, annotations of linguistic phenomena (should) follow a standard (e.g. @xcite . as has been observed by @xcite , there are still features that are missing or only seldomly addressed in annotation tools. so further developments are here to be expected. however, we argue that another technical and conceptual change takes place, a change that is characterised by the following, partly mutually influencing, features. ## related work there are applications around that address some of the features outlined above. we are aware of the following ones: incorporating machine learning applications into the annotation pipeline is carried out in inception (de castilho et al., 2019) (which extends on webanno (eckart de castilho et al., 2016)), the commercial service prodi.gy (montani and honnibal, 2018) and the textim-ager @xcite (the latter also underlies the present work). ## textannotator textannotator is a suitable candidate as a multimedia and multimodal annotation environment for uima documents @xcite . the uima-based annotations are driven by the tex-tannotator as a restfull application developed in java. documents which are not available in uima can be transferred into this format by using textimager @xcite , which provides a rich machine learning backend for automatic annotation accouting for the 'ml' component of muvamp. the uima documents are stored through the uimadatabaseinterface (abrami and mehler, 2018) within mongodb foot_0 and can be used simultaneously and collaboratively through tex-tannotator. collaborativity and simultaneity are enabled with bidirectional information exchange via web-socket between textannotator and all client systems and is an important component for ubiquitous use. in addition, the web-socket allows other annotation tools to be connected to textannotator, to ensure its multimedia nature (see sec. 4 and 5). the connection between tex-tannotator and its client systems is illustrated in fig. 2 . annotations stored in uima documents are organized in different annotation views (av). each of these views contains different annotations and is related to a specific topic or user. for each annotator, a user view is created when a document is initially opened, which duplicates the original annotations. thus, each av shows a different perspective, state, or context on the same document. furthermore, the different avs allow the computation of inter-annotator agreement, which enables to assess the consistency of annotations in a project @xcite , based on user permissions. in addition, all annotations can be used independently of textannotator: they can be completely downloaded for further processing. the reuse of the annotations as a basis for ml is thus customizable, depending on the needs of the particular application. in the following, we show how textannotator accounts for multi-mediality and -perspectivity, dynamics of annotation pro-cesses, and ml for annotation. vr and ar, and ubiquitous use are dealt with in sec. 4 and 5, respectively. quickannotator still allows rapid annotation of named entities and words and multi-token expression in general through a simple selection of a target class and subsequent assignment when clicking on tokens @xcite . to increase annotation performance, a recommendation function was implemented that allows the selected target class, based on the token's lemma, to be applied to all other tokens of the same lemma in the same document, paragraph, or current sentence. in addition, all tokens annotated by this function are marked so that annotators can easily target and post-process them. another function is the combination of tokens to multitokens: by now this function has been extended with the possibility to separate tokens at any position as well as the capability to correct ocr errors (see fig. 1 ). this user-friendly function, which can also be executed via drag & drop, enables the correction of incorrectly recognized token boundaries which is a frequent and popular error, especially in the context of ocr recognition of texts. as it provides basic corrections to the texts and establishes the prerequisites for future annotation processes, quickannotator develops into a preediting tool which is employed before the main annotation work is done with more specific tools such as propannotator or depannotator. hence, the triplet of quickannotator, propanno-tator, and depannotator gives rise to multi-figure 1 : tokens can be corrected as required. firstly, incorrect token boundaries can be split using a simple key combination. in the present example this was done with the token merged with the comma (green border). secondly, ocr errors can be corrected in quickan-notator by clicking on the corresponding tokens. in this way the original text is not changed, but a correction token is generated, which is placed on top of the affected tokens. a corrected token is visualized with a green dot in the upper right corner. by moving the cursor over it, the original token is shown. perspectivity (in particular if annotation files are preprocessed in such a way that they contain annotation layers according to the corresponding tools' specification). propannotator uses relations adopted from the semaf standard (iso, 2014a) as well as from prop-bank @xcite (the latter can be mapped onto the former @xcite ). to this end, we converted the structures defined in these standards into a uima type system foot_1 . since the last presentation of propannotator in @xcite , significant improvements and new features have been implemented. the underlying data model allows for annotating a wide range of relations a subset of which is available in the current web interface of propannotator: depannotator is the newest tool designed for visualization and annotation of dependency structures in texts. based on different dependency tag sets (derived from tiger @xcite , respectively negra @xcite , and universal dependencies (de marneffe et al., 2014)) existing dependencies can be deleted and new ones can be created. in addition, as with all tools, it is also possible to manually annotate texts without pre-annotated dependency information, which is illustrated in fig. 4 . ## mobileannotator to remove the binding of the annotation situation to desktop sessions, so to speak, to enable annotations in mobile contexts, quasi ubiquitously (whether in sitting, standing or walking position of the annotator), we have developed mo-bileannotator @xcite . based on angular 4 , we adapted two tools of textanno-tator (quickannotator and knowledge-baselinker) to enable mobile access. mo-bileannotator was developed as a textan-notator client (see fig. 2 ) using its functionality. this allows the implementation of additional functions which are not available in the browserbased version. at the same time, documents are still accessible only after user authentication and all annotations are stored in mobileannotator in appropriate annotation views. the the control and use of uima documents is thus analogous to textannotator. to motivate it with concrete examples: with mobileannotator, train rides, waiting time at the doctor's office, at the bus stop, or anywhere else can be used for annotation tasks. mobile annotations, of course, attain ubiquity. ## vannotator vr-based annotation is provided by vanno-tator, a uima-based annotation environment implemented in unity3d 5 . since vannota-tor @xcite is also based on 4 https://angular.io/ 5 https://unity.com/ textannotator (see fig 2 ), its annotations can be further processed with any other annotation media of textannotator. vannotator addresses a range of scenarios: visualization and interaction with historical information @xcite , annotation of texts, their interlinking with images and 3d objects, and the creation of 3d spaces enriched with hypertext functionalities @xcite . vannotator is currently extended to include semaf-related functionalities. a pilot study of this extension is presented in @xcite . the main focus is on the annotation of spatial relations (isospace, @xcite , semantic roles (srlinks, iso, 2014a) and coreference relations @xcite . this is done to generate text-to-scene data, which in turn is used to train ml systems. fig. 6 exemplifies this sort of annotation data. in this example, we take advantage of the spatial capabilities of vr to automate as many spatial annotations as possible. that is, whenever the annotator arranges objects in virtual space based on their description in the underlying text, a subset of the relationships of the objects implied by this arrangement is explicitly annotated by the system itself. this concerns objects that are implicitly or explicitly involved in figure 4 : the interface of depannotator is similar to that of propannotator. on the left one finds the sentences to be annotated (together with their annotation status). the panel on the right displays selectable options subdivided into tagsets. two visualizations of the focal sentence are displayed in the center of the window: the lower sentence shows the dependency tree created by the parser selected in textimager; the upper sentence shows its correction. green lines encode selected dependency relations created by the human annotator. with depannotator it is possible to visually compare automatically created dependency trees and their corrections. depannotator additionally contains statistics for automatic comparison of such trees. ## application usage the tools described so far are being used in various lectures and qualification work (e.g. kühn (2018); smaji (2020); kett (2020); lööck (2020) ) to automatically validate annotated documents (human in the loop) or to gain new perspectives on annotated documents. in particular, textannotator is used as a browser-based suite for the correction of automatic annotations generated with textim-ager. in addition, textannotator is used in the biodiversity project biofid @xcite . this project is concerned with the semantic indexing of historical biodiversity texts. for this purpose, textannotator is used to annotate texts in order to perform various linguistic analyses. within the biofid project, 79,813 "net" annotations foot_3 have been produced using quickannotator @xcite . "net" means the following: since within biofid documents are annotated by more than one annotator for the sake of assessing interrater agreement, one and the same annotation unit may receive a label repeatedly but from different annotators. the net count ignores such reduplications and only takes unique labels into account. these numbers show that many annotations can be performed by different users in a very short time (for a user evaluation of one of textannota-tor's modules, treeannotator, see @xcite . at the same time, all annotations are available in a uniform and portable format, which ultimately simplifies external processing and reuse, e.g., for ml tasks. the combination of a large number of different annotation functions (at the word, sentence, or text level) that provide multiple annotation perspectives on the same text, as well as the multimedia bandwidth that comes with them, is, to our knowledge, currently unique in the field of annotation of natural language texts. ## future work currently, not all annotation features available through textannotator can be used by downstream tools (e.g. mobileannotator). to enable full ubiquitous use, different approaches for the different media (vr, ar, mobile devices) are required. in particular, we will consider the possibilities and limitations of ar systems and the extent to which they can be used for annotation purposes. while the available hardware is still very limited (price, availability, technical features, . . . ), in the near future it will become available to the general public, similar to vr. furthermore, in addition to the extension of vannotator's roombuilder according to the semaf standards, an annotation environment for textannotator and mobileannotator is planned. this extension should make it possible to pre-annotate texts at home, in the office, or on the road, to largely complete their annotation in vr -also with regard to implied annotations -and to correct and refine the results later with conventional 2d interfaces if necessary. insofar as these annotations refer to artifacts that are visible or even traversable in reality (e.g. streets, houses, squares), this multimedia annotation process can be significantly enriched by ar functionalities, since the direct view of the objects to be annotated can compensate for inadequacies of their representation in vr. thanks to the large number of tools in textan-notator, a wide range of annotation tasks can be addressed. however, since it is inefficient in the long run to develop tools with reference to specific annotation requirements, a more dynamic approach that simplifies the planning of annotation projects suggests itself. to meet this requirement, textan-notator is being further developed as a tool for modeling annotation models and corresponding annotation tools. furthermore, it is planned to publish textannotator via github. ## conclusion we introduced the concept of muvamp (multimediality and -perspectivity, ubiquity, vr/ar, ml, process-orientation) and argued how textannotator and the annotation tools around it meet this concept. reflecting on and studying muvamp, and devising corresponding annotation models is still a desideratum for computational linguistics. the increasing complexity of annotation tasks and their representation in tools in order to be able to use them collaboratively and simultaneously in a homogeneous annotation environment at best. in addition, enabling annotators to use multi-perspective multimedia annotation tools is an area where established best practices do not yet exist. in order to contribute to this research perspective, we have presented the latest developments of textannotator and outlined future development steps. in conclusion, it is our strong interest to discuss and also establish with the research community a new and more innovative way in the implementation of annotation processes. for this purpose, not only concepts and procedures are necessary, but also adequate and flexible software solutionssuch as textannotator.