The D - WISE Tool Suite: Multi-Modal Machine-Learning-Powered Tools Supporting and Enhancing Digital Discourse Analysis (2023)

Abstract: This work introduces the D-WISE Tool Suite (DWTS), a novel working environment for digital qualitative discourse analysis in the Digital Humanities (DH). The DWTS addresses limitations of current DH tools induced by the ever-increasing amount of heterogeneous, unstructured, and multi-modal data in which the discourses of contemporary societies are encoded. To provide meaningful insights from such data, our system leverages and combines state-of-the-art machine learning technologies from Natural Language Processing and Com-puter Vision. Further, the DWTS is conceived and developed by an interdisciplinary team ofcultural anthropologists and computer scientists to ensure the tool’s usability for modernDH research. Central features of the DWTS are: a) import of multi-modal data like text, image, audio, and video b) preprocessing pipelines for automatic annotations c) lexical and semantic search of documents d) manual span, bounding box, time-span, and frame annotations e) documentation of the research process.

## introduction in today's digital era, ever-increasing amounts of heterogeneous, unstructured, and multi-modal data are ubiquitous. within this data, discourses of contemporary societies are included in various forms, such as news articles or videos, social media postings, forum threads, memes, podcasts, or tv shows. this induces an issue for digital humanities (dh) researchers when conducting digital qualitative discourse analysis @xcite with such data to examine complex sociological patterns and discussions: it becomes infeasible for a researcher or an average research group to investigate the data manually so they rely on computer assisted qualitative data analysis software @xcite . although there are many such tools @xcite @xcite , they often lack support for such (amounts of) data or are proprietary software. with the d-wise tool suite (dwts) introduced in this work, we provide a novel working environment to support and enhance digital qualitative discourse analysis. the tool is conceived and developed within the d-wise foot_0 project in close co-creation by an interdisciplinary team of cultural anthropologists and computer scientists to ensure the tool's usability for modern dh research. our system relies on recent advances in natural language processing and computer vision and their combination to address the challenges of large amounts of heterogeneous and multi-modal data. specifically, we employ state-of-the-art text @xcite , vision @xcite , speech @xcite , video @xcite , and visio-linguistic models @xcite in our multi-modal preprocessing pipeline for tasks like named entity recognition, multi-modal similarity search, object detection, image captioning, automatic speech recognition (transcription), and video understanding tasks. other essential functionalities for digital discourse analysis like lexical search or manual annotations, are also supported by the dwts. in this paper, we describe the system architecture and central features of the dwts: (a) import of text, image, audio, and video documents; (b) preprocessing pipelines for automatic annotations and indexing; (c) lexical and semantic similarity search, (c) manual annotations; (d) automatic and manual documentation of the research process. the dwts is designed as an extensible and scaleable opensource software system and is publicly available on github foot_1 . links to a demonstration instance foot_2 and a video foot_3 are provided. ## related work in the following, we discuss the features and limitations of popular open-source caqdas and general purpose annotation tools concerning qualitative discourse analysis on large heterogenous, multimodal, and multi-lingual document collections. webanno @xcite , codeanno @xcite , incep-tion @xcite are a family of web-based collaborative platforms for linguistic annotations based on the uima framework. while they support a wide range of annotation-related features for text documents, they do not support image, audio, or video material. although there exists a webanno extension @xcite for multimodal document annotations, the tool's objective is annotating relatively small corpora. further, the platforms do not include any deep learning technology or preprocessing steps for automatic annotations. moreover, all platforms lack an overview of the document collection and general search functionality, making it challenging to apply for discourse analysis projects. another popular tool is catma @xcite . although the webbased platform is designed for qualitative and quantitative research, including annotation, analysis, and visualization functionalities, it only supports text documents. other caqdas systems involving deep learning techniques for automatic annotations are labelsleuth @xcite and textannotator @xcite . however, both tools only support textual documents. recogito2 @xcite is an open-source tool that can be used for collaborative research projects, is applicable for discourse analysis, and supports text and image annotations. however, the tool does not implement preprocessing pipelines and is designed for small document collections. the widely used caqdas tools maxqda and atlas.ti fulfill almost all considered criteria but are proprietary closed-source software, and require expensive licenses, what often poses a hurdle of academic research groups. further, the tools do not support advanced deep learning features like similarity search, object detection, or automatic audio and video transcriptions. ## system demonstration the d-wise tool suite is a web-based working environment for digital qualitative discourse analysis. it is designed to handle multi-lingual, multi-modal material of large corpora, achieved by a scaleable architecture built upon a unified data model and state-of-the-art machine learning technology. the application is developed for research groups featuring collaborative projects with multiple users and role management. further, the system is easily deployable using docker, not requiring the installation of additional third-party software. generated data can be exported in common formats for further analysis or processing with other tools. ## system architecture the dwts is a client-server application with a python backend and a react frontend. a rest api encapsulates core functionalities to enable communication between frontend and backend. ## conclusion this paper presented the d-wise tool suite (dwts), a web-based open-source application to support and enhance digital discourse analysis for digital humanities by being able to operate on large amounts of heterogeneous and multi-modal data. we discussed the motivation and need for our system by pointing out the limitations of existing dh tools for extensive multi-modal document collections. further, we demonstrated central functionalities of the dwts by describing a typical workflow illustrated by screenshots, and provide technical details about the system architecture in a separate section. currently work-in-process but released in future work are video and audio annotations and 4-way multi-modal similarity search between text, image, audio, and video documents.