A nno P lot: Interactive Visualizations of Text Annotations (2024)

Abstract: This paper presents AnnoPlot, a web application designed to analyze, manage, and visualize annotated text data.Users can configure projects, upload datasets, and explore their data through interactive visualization of span annotations with scatter plots, clusters, and statistics. AnnoPlot supports various transformer models to compute high-dimensional embeddings of text annotations and utilizes dimensionality reduction algorithms to offer users a novel 2D view of their datasets.A dynamic approach to dimensionality reduction allows users to adjust visualizations in real-time, facilitating category reorganization and error identification. The proposed application is open-source, promoting transparency and user control.Especially suited for the Digital Humanities, AnnoPlot offers a novel solution to address challenges in dynamic annotation datasets, empowering users to enhance data integrity and adapt to evolving categorizations.

## introduction within the context of nlp, sequence annotation is the practice of tagging sections of text with appropriate labels. for example, this could involve classifying the phrase "finding nemo" as a "movie" in the sentence "i love finding nemo." annotation projects, such as the creation of annotated datasets, face a variety of challenges, one of which is data quality and validity. as data is added and use cases evolve, prior category definitions might be insufficient and need to be updated or reorganized. as datasets grow and evolve, achieving a good overview becomes increasingly complex, which is necessary to revise labeling strategy and category definition @xcite . annoplot addresses this issue by offering comprehensive two-dimensional views of the provided dataset without any prior training or classification model necessary. such views of span annotations and category systems enable users to: annoplot generates two-dimensional representations akin to systems such as bertopic @xcite . first, the span annotations are encoded with a pre-trained transformer model, yielding large embeddings, typically of 512 up to 1024 dimensions. in this work, we use bertbased transformer encoders since bert @xcite embeddings have proven to be very information-rich, meaning a large variety of data is encoded in the high-dimension embeddings. next, dimensionality reduction techniques are implemented to generate a two-dimensional representation for visualization purposes. our proposed tool offers an interactive dimension reduction algorithm based on the well-known umap @xcite algorithm and its parametric neural network implementation @xcite . the user can remotely train the small reduction algorithm network to fit annotated text passages to the categories. in case of further customization necessity, dots or clusters representing annotations can be dragged in the interactive view to inform the model about the desired changes for the subsequent training iterations. this training happens in real-time; the user can see updates or stop the process in case of mistakes or overfitting. annoplot provides a customizable and dynamic application for the analysis of annotated datasets that stands apart from expensive commercial solutions like maxqda or galileo. our application lets users interactively modify annotations, hierarchical category systems, and visualizations in real-time without requiring pre-trained, categoryspecific models. by bridging the gap between advanced data visualization techniques and userfriendly applications, annoplot makes a noteworthy addition to current annotation tools. annoplot is an open source project; the code can be found in its github repository foot_0 together with a short video demonstrating its core functions. a live demonstration of annoplot can be found here foot_1 . ## related work annoplot offers an interactive user interface for data visualization of annotated texts and classification category management. in the context of category structuring and reorganization, annoplot can be compared with maxqda's creative coding feature. maxqda foot_2 , a qualitative data analysis software, facilitates the organization and hierarchical structuring of codes through its creative coding functionality. this process involves generating, sorting, and organizing codes, defining relationships between them, and creating a hierarchical structure. however, creative coding primarily focuses on manual categorization and organization and does not offer any automation. in contrast, annoplot leverages embedding representations to offer an initial starting point while offering similar category management functionality. examining general visualization, annoplot's 2d visualization of annotated text passages can be contrasted with tools like the embedding view functionality of galileo foot_3 . galileo is a platform that provides tools and modules to evaluate machine learning classifiers, including nlp classifiers for sequence annotation. the embedding view utilizes embeddings and a parametric umap reduction to visualize classifier results. however, its primary focus is on showcasing classifier performance, and potential areas of uncertainty, rather than facilitating dataset analysis or category reorganization. annoplot, on the other hand, is designed for quick and interactive analysis of annotated text datasets, enabling users to gain insights into the overarching structure and quality of their annotations. the absence of a classifier in annoplot makes its interactive dimension reduction feature particularly valuable, allowing for real-time adjustments and customization based on user input. in terms of visualization techniques that utilize a processing pipeline of embedding and dimensionality reduction, annoplot shares similarities with many applications such as bertopic (grootendorst, 2022) and the tensorflow embedding projector @xcite . in regards to user interactions, research analysis of interactive dimension reducing tools @xcite identified seven distinct categories of interactive dimensionality reduction algorithms, three of which apply to annoplot: a notably similar approach to interactive data visualization is taken by zexplorer @xcite an extension of the bibliography system zotero. zexplorer maps selected features of research papers, including a bert embedding of the paper's abstract, to two dimensions utilizing a neural network approximation of umap. similar to annoplot it offers interactive training of this network through draggable data points. the development of annoplot was motivated by the absence of interactive features for the visual analysis of annotations in the most popular @xcite open-source annotation tools such as webanno @xcite , brat @xcite , catma @xcite , and in-ception @xcite . annoplot combines contextualized embedding representations of annotations with interactive visualizations for analyzing annotated datasets and category structuring. ## system architecture the application is designed to visualize annotated text segments within the provided datasets and facilitate the modification of existing annotations. further, it computes and assigns clusters to text segments, helping users to identify annotation errors. it allows users to create projects to manage datasets within it and the flexibility to import data in standard data formats such as conll @xcite . when exporting the data, datasets within a project are merged. the tool includes a search function to navigate the dataset and the clusters. finally, the tool offers detailed statistics for projects, codes, and clusters. annoplot offers two views: one dedicated to presenting annotated text segments and the other focused on hierarchical category systems. users can create, delete, and rename data points in both views. furthermore, users can merge categories and train the existing embeddings further. the views are rendered using the d3.js library. in general, the frontend implementation of the application is realized through react/next.js. the backend was implemented in python using fastapi and postgresql for data storage. the data processing pipeline to compute 2d representations of annotated text passages consists of a bertbased transformer model, umap, and hdbscan. the hugging face library is employed to load models and allows for flexibility in which model to use in the pipeline. the hardware requirements for running anno-plot are largely dependent on the embedding model chosen. for development and testing, we used the "bert-base" model with a 3080 rtx gpu (10 gb). the backend uses transformer/bert-based models for inference and, at most, trains a small (4 layer) dynamic umap model. while a gpu is not strictly required it is highly recommended for ux performance. annoplot is deployed using docker. every system component i.e., the database, frontend, and backend is deployed in a separate docker container, which are orchestrated in a docker compose file. ## evaluation to evaluate the effectiveness of error detection and overall clustering validity, errors were intentionally introduced to a dataset. the tool was then monitored on its ability to identify these via its 'error detection' feature. ## conclusion & future work in this paper, we presented annoplot, a web application that allows users to easily visualize, manage, and analyze annotated datasets. the proposed tool offers an efficient and streamlined solution for working with span annotations and their category system in a novel way on a 2d canvas. it enables users to upload datasets, add configurations, and manage them as projects, with different data extracted and presented across separate pages. the category and plot views are the main features, displaying categories and annotations clustered together. users can perform tasks like merging, deleting, and rearranging annotated text passages directly within the view. the plot view also enables the training of dynamic umap to optimize the visualization and identify any errors or outliers in the clusters. statistics are available for categories, clusters, and projects. this work focused on the visualization, management, and analysis of annotated text datasets. however, extending the proposed approach and interface to support annotated data of other modalities is straightforward. in a future iteration of this tool, we will support image annotations by switching the embedding model with appropriate visual transformers and adapting the software to handle image datasets. this broadens our tool's analytical scope to encompass a large variety of data types. further, annoplot will offer more varied dimensionality reduction algorithms, including t-sne and principal component analysis. finally, it is planned to integrate annoplot into a more comprehensive open-source annotation software as an interactive visual analysis feature.