Semi-automatic Sequential Sentence Classification in the Discourse Analysis Tool Suite (2025)

Abstract: This paper explores an AI-assisted approach to sequential sentence annotation designed to enhance qualitative data analysis (QDA) workflows within the open-source Discourse Analysis Tool Suite (DATS) developed at our university.We introduce a three-phase Annotation Assistant that leverages the capabilities of large language models (LLMs) to assist researchers during annotation.Based on the number of annotations, the assistant employs zero-shot prompting, few-shot prompting, or fine-tuned models to provide the best suggestions.To evaluate this approach, we construct a benchmark with five diverse datasets.We assess the performance of three prominent open-source LLMs — Llama 3.1, Gemma 2, and Mistral NeMo — and a sequence tagging model based on SentenceTransformers.Our findings demonstrate the effectiveness of our approach, with performance improving as the number of annotated examples increases. Consequently, we implemented the Annotation Assistant within DATS and report the implementation details.With this, we hope to contribute to a novel AI-assisted workflow and further democratize access to AI for qualitative data analysis.

## introduction the discourse analysis tool suite (dats) @xcite is a platform developed at our university to empower digital humanities (dh) researchers in conducting qualitative data analysis (qda). developed collaboratively and tailored to the specific needs of dh scholars, the platform democratizes access to machine learning methods. it enables non-experts to effectively manage and analyze large-scale, unstructured, multi-modal data. while the platform's overarching design is rooted in grounded theory-based research @xcite , its versatile features support various disciplines. key functionalities encompass automated pre-processing of multi-modal data (text, image, audio, and video), comprehensive data exploration capabilities, and diverse quantitative analysis tools. one of the core qda tasks on our platform is in-depth qualitative analysis through the manual annotation of text documents. this process involves searching for documents relevant to the users' research question, creating and extending a category system (taxonomy), and diving into the material to annotate relevant text passages. while dats currently supports span-level annotations (similar to named entity recognition), we observed during collaborations with researchers in various disciplines that they primarily annotate at the sentence or paragraph level (i.e., annotating a sequence of sentences). moreover, they often want to analyze the distribution of their taxonomy across the entire dataset -for example, to see how a particular category usually occurs or how categories relate to each other. however, obtaining such quantitative insights requires annotating a large amount of data, which is often impractical to do manually. to address this need, this work aims to support sequential sentence annotation within dats and provide researchers with tools for efficient and practical analysis. in collaboration with our project partners, we have developed a new user interface specifically designed for sequential sentence annotation. here, a single sentence is the smallest possible unit to annotate. this interface includes functionalities for comparing annotations between users and, most importantly, an annotation assistant that learns from existing annotations to provide suggestions for unseen data. central to our approach is a three-phase annotation assistant designed to provide progressively refined suggestions as users annotate data. key requirements drive this design, including user control over the annotation process, optimal suggestions at each stage, and fast inference for a smooth user experience. the three phases are: (1) zero-shot prompting: when no annotations are available, the assistant leverages the taxonomy and definitions to generate initial suggestions via zero-shot prompting of large language models (llms). (2) few-shot prompting: as users annotate data, the assistant transitions to few-shot prompting, refining its suggestions by incorporating user-provided examples. (3) fine-tuned model: with sufficient annotations, a sequence tagger can be fine-tuned to this data, further improving the accuracy of suggestions. to evaluate the effectiveness of our three-phase approach and choose the best model to integrate into the dats, we construct a benchmark consisting of five datasets on sequential sentence classification. we assess the zero-and few-shot performance of three prominent open-source llms: llama 3.1, gemma 2, and mistral nemo. further, we fully fine-tune and evaluate sequence tagging models on this benchmark. our approach works best with mistral nemo, steadily improving with an increasing number of annotated samples. the contributions of this paper are: recognized for qualitative and mixed-methods research, maxqda foot_0 has introduced "maxqda ai assist," offering ai-driven features like summarization, paraphrasing, concept explanation, and automatic transcription. recently, "ai coding" was made available as a beta feature that assists with annotating single documents. nvivo foot_1 , a platform for qualitative data analysis, integrates ai features in its latest beta version, including thematic coding, sentiment analysis, and text summarization. known for visual and network analysis tools, atlas.ti foot_2 integrates openai's gpt models with existing features like code suggestions, sentiment analysis, summarization, and entity recognition. remarkably, ai-powered features are only found in paid versions of such qda platforms. further, none of these tools currently offer functionalities to train models and automatically analyze large corpora for quantitative insights. in contrast, dats aims to democratize access to state-of-the-art ai capabilities, making advanced functionalities such as the proposed three-phase annotation assistance freely available to researchers across disciplines. to avoid data protection issues often associated with cloud-based ai services, dats can be run in-house without sending sensitive research data to third-party providers. benchmarking several general llm benchmarks like mmlu @xcite , su-perglue @xcite , big-bench (srivastava et al., 2023), helm @xcite , and mteb @xcite have emerged. while they cover many tasks, they may not necessarily be the most relevant for qda. @xcite evaluates llms' zero-shot performance on a range of computational social science (css) tasks focusing on taxonomic labeling and free-form coding. their experiments indicate llms' great potential to augment css research as zero-shot data annotators, strongly motivating our approach to using llms in zero-and few-shot scenarios. our benchmark evaluates sequential sentence classification across various domains and tasks related to css and informs the integration of our proposed annotation assistant into dats. ## envisioned workflow inspired by project partners who actively work with the discourse analysis tool suite, this section describes an illustrative data analysis workflow highlighting potential areas where the annotation assistant could enhance productivity. imagine alice, a social science researcher investigating the climate change debate in germany. she is interested in understanding how different news outlets frame climate activism. to begin her research, alice gathers a large dataset of relevant news articles and uploads them to dats. before annotating, alice and her project team brainstorm a taxonomy of frames relevant to climate activism. the taxonomy includes frames like "economic," "health," "fairness and equality," and others. they carefully define each frame and establish annotation guidelines. with the taxonomy in place, alice opens dats' new sentence annotator (see figure 1 ), eager to start labeling her data. but even before she annotates a single sentence, the annotation assistant is ready to help. being provided with the taxonomy and its definitions, the assistant can offer initial suggestions. alice clicks the robot button (1) to activate the assistant, selects the frames she wants to focus on, and initiates the process. a side-by-side view appears, displaying alice's annotations on the left and the assistant's suggestions on the right. alice reviews these suggestions: hovering over an annotation highlights the corresponding text (2). hovering over the category name in the taxonomy highlights all annotated texts. she accepts the correct annotations with the help of the control buttons in the middle (3). she can easily create annotations by clicking and dragging sentences where needed. after alice annotates a few documents and reaches a certain threshold of annotations per frame, the assistant leverages these to refine its suggestions through few-shot prompting. the improved suggestions accelerate alice's progress, allowing her and her team to annotate a significant portion of the dataset efficiently. with a larger pool of annotated data, the assistant fine-tunes a specialized model on these examples, further improving the quality of its suggestions. alice and her team begin accepting suggestions in batches, significantly boosting their productivity. confident in the assistant's capabilities, they let it automatically annotate the remaining documents. this process takes time. the next day, alice reviews some of the assistant's annotations. satisfied with the quality, she proceeds to quantitative analysis. for example, she can utilize dats' built-in features to visualize how climate change framing evolves over time or export the annotated data for further analysis with her preferred tools. ## semi-automatic sequential sentence classification the workflow described in the previous section highlighted how the annotation assistant could empower researchers like alice to analyze large textual datasets effectively. to achieve this, we designed the assistant with several key requirements in mind, focusing on user-centricity: based on the requirements, we propose a threephase approach to ai-assisted annotation: phase 1: zero-shot prompting when no annotations are available, the assistant leverages the provided taxonomy and definitions to generate initial suggestions through zero-shot prompting of llms. please note that the user provides the taxonomy and category definitions, which are fully customizable. phase 2: few-shot prompting after the user annotates a few documents and meets a certain threshold of labeled examples per category, the assistant transitions to few-shot prompting, refining its suggestions by incorporating the user-provided examples. phase 3: fine-tuned model once sufficient annotations are available, a sequence tagger is finetuned on this data, further improving the accuracy of the assistant's suggestions. notably, the thresholds for transitioning between phases (e.g., what constitutes "few" or "sufficient" annotations) are configurable. they can be adjusted based on the specific characteristics of the dataset and the user's preferences. further, the reasoning is currently only supported during phase 1 and 2, as the llm is instructed to provide a reason alongside its prediction. ## benchmarking sequential sentence classification this benchmark evaluates llms and sequence taggers on sequential sentence classification tasks. as outlined in the previous section, both are crucial components of our proposed three-phase approach. we aim to identify the most suitable configuration for integrating practical annotation assistance into dats. to this end, we carefully select datasets for sequential sentence classification with varying domains and tasks. sequential sentence classification is a sequence tagging task similar to, for example, named entity recognition, but instead of tagging tokens, sentences are tagged. hence, every sample of a typical sequential sentence classification task consists of a list of sentences and tags assigned to each sentence. in this benchmark, we only investigate the single-label classification task, i.e., every sentence is classified into one of many classes. ## integration within dats the annotation assistant is built using react for the frontend and ollama, fastapi, and celery foot_3 for the backend. celery handles background job processing, including fine-tuning and inference of the sequence tagger model. it ensures that assistance tasks run without interrupting user workflow. ollama hosts the mistral nemo model. it also handles the structured generation in combination with pydantic. we reuse the prompts utilized in the benchmark. dats already offers a semantic similarity search based on sentence embeddings, which are computed during import. we reuse these pre-computed sentence embeddings to significantly reduce the training and inference time of the sequence tagger model. ## conclusion this paper explored a three-phase approach to aiassisted annotation to enhance qualitative data analysis workflows, focusing on sequential sentence classification within our open-source discourse analysis tool suite. we designed a benchmark and evaluated the performance of three prominent opensource llms and sequence tagging models based on sentencetransformers. our findings demonstrate the effectiveness of our proposed three-phase annotation assistant. performance steadily improves as more annotated data becomes available when using mistral. however, few-shot prompting does not scale as well as expected and may require a different approach. we integrated this assistant into dats, providing researchers with a valuable semi-automatic sequential sentence annotation tool. we see this integration as a significant step towards democratizing access to ai for data analysis, empowering researchers with a transparent and user-controlled system that augments, rather than replaces, their expertise. in future work, we plan to investigate parameterefficient fine-tuning of llms to bridge the performance gap observed between few-shot prompting and full fine-tuning. additionally, we aim to expand the benchmark to include german datasets, catering to dats's user base that works with german-language texts. code for replicating the benchmark foot_4 , the repository of dats foot_5 and a live demo are available foot_6 .