Towards Unified, Dynamic and Annotation-based Visualisations and Exploration of Annotated Big Data Corpora with the Help of Unified Corpus Explorer (2025)

Abstract: The annotation and exploration of large text corpora, both automatic and manual, presents significant challenges across multiple disciplines, including linguistics, digital humanities, biology, and legal science. These challenges are exacerbated by the heterogeneity of processing methods, which complicates corpus visualization, interaction, and integration. To address these issues, we introduce the Unified Corpus Explorer (UCE), a standardized, dockerized, open-source and dynamic Natural Language Processing (NLP) application designed for flexible and scalable corpus navigation. Herein, UCE utilizes the UIMA format for NLP annotations as a standardized input, constructing interfaces and features around those annotations while dynamically adapting to the corpora and their extracted annotations. We evaluate UCE based on a user study and demonstrate its versatility as a corpus explorer based on generative AI.

## introduction the automatic processing and manual annotation of large text corpora poses a general challenge for various projects in different disciplines such as linguistics (e.g. @xcite ; @xcite ) literary science (e.g. @xcite ; @xcite ), digital humanities (e.g. @xcite ; jiménez-badillo et al. (2020)), biology (e.g. @xcite ; @xcite ), chemistry (e.g. @xcite ; @xcite ), biodiversity (e.g. @xcite ; @xcite ) and legal science (e.g. @xcite ; @xcite ), to name a few. in addition to a multitude of processing methods and the underlying representation formats, which are not directly interoperable due to their mutual heterogeneity @xcite , one more challenge arises: the flexible search and interactive visualization of search results extracted from corpora processed by any of these nlp tools. this challenge can be attributed to high computational demands based on corpus size, the absence of general-purpose tools with built-in interaction features, and the need for standardized formats or pipelines. we address these challenges by means of uni-fied corpus explorer (uce) (section 3), which we developed to facilitate corpus navigation and exploration (section 3.2) using generic, uimabased @xcite ) nlp pipelines. in this context, the "unstructured information management architecture" (uima) ensures data interoperability through its flexible xml metadata interchange (xmi) schema-based annotation format. additionally, uce is standardized in the sense of being dockerized and reusable (section 3.1), ensuring broad applicability across various domains (section 4). the standardized and systematic creation of uce is accomplished by using the docker unified uima interface (duui - @xcite ), a framework for the distributed and scalable processing of nlp tasks (section 3.3). to evaluate uce, we conduct a user study (section 5) that demonstrates its versatility in the context of information retrieval based on parliamentary debate corpus data. finally, we release uce as an open-source framework foot_0 (agpl-3.0 license) with a public web demonstration foot_1 and an online screencast foot_2 . ## related work visualizing and interrelating textual information in a functional, appealing, dynamic, and interactive way is a challenge in many disciplines and projects. a web-based tool for this is reden on-line @xcite which, in the context of literary projects, allows the generation and visualization of similarity relations via a web interface using tei-based editions. another tei-based approach is to visualize shakespeare's plays @xcite . interhist @xcite provides explorations of the paisà corpus @xcite to support linguistic projects. the corpus can be searched and results can be visualized as charts. a non-browser-based application focusing on visualizing relations within social groups is provided by @xcite , who explore emotions to informationally enrich visualizations. another approach using nlp methods that is similar to uce, but does not include chatbots is the dvw project @xcite . it uses me-diawikis to represent the information explored by nlp, to integrate the associated annotations, and to visualize statistics using e.g. maps. since the underlying preprocessing software has been further developed, this project is no longer usable without adaptations. another web-based tool is annoplot @xcite ; it allows users to upload annotated data, analyze it, and generate scatter plots, clusterings, and text statistics. this includes embeddings that are reduced by a dimensionality reduction algorithm and used for visualization. with a focus on preprocessing based on nltk @xcite , webnlp @xcite offers various visualizations (e.g. word clouds) using voyant @xcite . as an example of llm-enhanced annotation, there is itake @xcite , which uses generative ai to speed up the process of annotating and extracting data from single texts (rather than retrieving information from many texts). these projects show that the functionalities of comprehensive corpus visualization and exploration (e.g., dynamic visualization of unstructured data, integration of ontological information, and semantic search based on llm-enhanced chatbots) are not yet sufficiently available. to fill this gap, we present uce, whose usage scenario can be outlined as follows: a scientist in a specific domain (e.g. chemistry or biology) has the task of gaining a detailed understanding of the content of a large number of texts. to do this, they must be able to skim the texts very quickly on an abstract visual level, as well as to find the smallest semantic units (e.g. at the level of semantic roles or individual arguments). this combination of general corpusrelated and detailed text-related information is done by uce, with the help of llm-based chatbots. ## unified corpus explorer we introduce a novel solution for making uimaannotated @xcite ) corpora tangible through our unified corpus explorer (uce). uce is a generic interface that, given any corpus and its extracted uima-based annotations, makes the underlying data accessible through various features, including semantic search and visualization, while integrating various uima-based annotations. uce imports the necessary files, creates a multi-microservice environment, and adapts to the needs of the corpus and its annotations. configuration files can be used to customize uce in terms of appearance (including color schemes and corporate identity), selection of active features, and integration of annotations. it also allows the incorporation of multiple corpora into the same instance. ## evaluation for evaluating uce, the use case from the area of parliamentary debates was chosen by selecting a sample from gerparcor @xcite , the largest collection of transcripts of german plenary debates annotated with spacy and sentiments. in order to perform evaluations in uce for the following scenarios, the existing annotations were extended by semantic role labeling using a duui-component based on @xcite : boxplot of the umux questions @xcite . the y-axis represents a 7-step likert scale, where 1 corresponds to "absolutely disagree" and 7 corresponds to "absolutely agree". a and b denote the split evaluator groups. the questions on the x-axis are to be read from left to right: 1: "the functionality of uce meets the requirements of the task at hand.", 2: "using uce is a frustrating experience.", 3: "uce is easy to use.", 4: "i had to spend too much time correcting issues in uce." 5: "the loading times were short." ## conclusion we introduced the unified corpus explorer (uce), a generic nlp system for exploring uimaannotated corpora. uce unifies the heterogeneous nlp landscape by providing a standardized framework that facilitates the collection, annotation, extraction, and visualization of large corpora in a customizable pipeline. its microservices-based architecture, implemented in docker containers, integrates multiple technologies and supports a wide range of annotations. this makes uce adaptable to various domains and research areas. our evaluation shows that uce provides a platform for addressing research questions using large corpora, but also identifies areas for improvement. future work includes improved filtering and reading capabilities, and better alignment of document structures with corpus specifics. scalability for large datasets requires stress testing of uce, while systematic evaluations of its tools (e.g. ccc-bert) are needed to assess the impact of uce on research projects. as its chatbot has proven to be a reliable tool, generative ai offers opportunities to improve uce. from a dh perspective, integrating wikidition technologies is also a future prospect. finally, explanatory and onboarding materials are needed to address usability issues identified in the evaluation.