Concept Over Time Analysis: Unveiling Temporal Patterns for Qualitative Data Analysis (2024)

Abstract: In this system demonstration paper, we present the Concept Over Time Analysis extension for the Discourse Analysis Tool Suite.The proposed tool empowers users to define, refine, and visualize their concepts of interest within an interactive interface. Adhering to the Human-in-the-loop paradigm, users can give feedback through sentence annotations. Utilizing few-shot sentence classification, the system employs Sentence Transformers to compute representations of sentences and concepts. Through an iterative process involving semantic similarity searches, sentence annotation, and fine-tuning with contrastive data, the model continuously refines, providing users with enhanced analysis outcomes. The final output is a timeline visualization of sentences classified to concepts. Especially suited for the Digital Humanities, Concept Over Time Analysis serves as a valuable tool for qualitative data analysis within extensive datasets. The chronological overview of concepts enables researchers to uncover patterns, trends, and shifts in discourse over time.

## introduction the discourse analysis tool suite (dats) @xcite serves as an open-source operational platform designed for conducting digital qualitative discourse analysis within the realm of digital humanities (dh). developed through collaborative efforts, the tool is tailored specifically for dh researchers. its purpose is to democratize access to cutting-edge machine learning technologies derived from computer vision and natural language processing. this initiative enables nonexpert users to efficiently handle and analyze unstructured, large multi-modal data. while the overarching design of the platform is catered to grounded theory-based research @xcite , particularly in alignment with approaches like the sociology of knowledge approach to discourse @xcite ), numer-ous features extend its utility across various disciplines. noteworthy functionalities include the automated pre-processing of multi-modal data (text, image, audio, and video), data exploration, and both manual and automatic annotation. in this paper, we extend dats with concept over time analysis, a machine-learning-based feature to identify concepts of interest in a large dataset and analyse their occurrences over time. the proposed system allows users to define concepts, assists them in identifying relevant sentences, refine the concept representation and finally visualize the results over time. the interactive user interface (ui) follows the human-in-the-loop paradigm @xcite , encouraging users to give feedback by annotating samples, which improves concept representations and analysis results. the proposed system performs few-shot sentence classification, mapping each concept to a class. we use sentence transformers @xcite to compute representations of sentences and concept descriptions. initially, a similarity search provides semantically relevant sentences for each concept that can be classified into concepts by the user. with few labeled examples, a contrastive dataset is generated to fine-tune a sentence transformer model. this is employed to update concept representations and similarity search results, providing the user with better suggestions. this iterative refinement step is repeated until the user is content. finally, sentences classified to a concept with confidence above a specified threshold are visualized in a timeline. the concepts over time analysis enables qualitative data analysis within large datasets. a chronological overview of custom concepts like events, topics, opinions or practices allows researchers to identify patterns, trends, and shifts in discourse over time. this temporal perspective can reveal how ideas, narratives, or cultural practices evolve, providing insights into the dynamics of discour-sive trends. by visualising such concepts along a timeline, researchers can identify clusters of related events, helping to create a nuanced understanding of how cultural phenomena unfold over time. further, a timeline analysis can aid in contextualising and connecting different elements within a large dataset, making it easier to trace the lineage of ideas or the influence of past discourses on current ones. though inspired by dh and discourse analysis, the proposed system is versatile, extending its utility to related analyses such as visualizing topics or sentiments over time. while the concepts over time analysis is a powerful feature on its own, its integration within dats can enable users to merge findings with existing analyses, utilize memos or the logbook for documentation and reflection, and identify concepts to be analyzed using search and filter functionalities. the contributions of this paper are threefold: first, we develop the concept over time analysis extension, which facilitates the identification of concepts in large datasets and a chronological overview. second, we describe a typical usage scenario, outlining the interaction with the system. finally, we perform experiments on three datasets, demonstrating the feasibility of our system. the dats and concepts over time analysis are open-source and available on github foot_0 where links to a demonstration and a video are provided. ## related work qualitative data analysis tools that are frequently used in the dh include catma @xcite , maxqda and atlas.ti. while offering various analysis tools, at the time of writing, none of them include means to semi-automatically identify concepts in large datasets or visualize concepts in a chronological overview. with the proposed extensions, dats aims to close that gap. however, there exist many tools and libraries that offer interesting timeline analysis functionalities. bertopic @xcite , a topic modeling tool, leverages sentence transformers to compute document or sentence embeddings, yielding easily interpretable topic representations through clustering. with dynamic topic modeling, the tool offers a collection of techniques to analyse the development of topics over time. while bertopic identifies prevalent topics automatically, our extension empowers users to define and analyze their own concepts or topics of interest across time. scot @xcite is an interactive web application to analyse the sense-clusters of a word and their evolution over time. sentiview @xcite is an interactive visualization system for sentiment analysis. it visualizes changes over time of various attributes and relationships between demographics of interest as well as participants' sentiments on popular topics. open discourse is a web-platform for the analysis of the plenary minutes of the german federal parliament. it includes a topic analysis feature to investigate the political discourse over time, filterable by speaker, gender, party and other attributes. labelsleuth @xcite serves as a nocode platform, making nlp accessible for a broad audience. it facilitates integrated model training and an intuitive, active learning-powered annotation interface. in contrast, our proposed extension supports the training of multi-class sentence classification models and enables a timeline analysis of the results. ## concept over time analysis the concept over time analysis extension is an interactive, machine-learning-based tool that semiautomatically identifies relevant sentences of userdefined concepts, adjusts accordingly to feedback, and visualizes concept occurrences in a timeline. its goal is to provide users with a chronological overview of their concepts of interest, with as few steps as possible. integrating the feedback process into the workflow was a key requirement for the development of the ui. ## system implementation the visualization is implemented using react and the recharts foot_1 library. the asynchronous model training is implemented with celery foot_2 and set-fit foot_3 @xcite . the main challenges involve classifying concepts and adapting to limited user feedback while ensuring high performance and interactivity. additionally, we noted that concepts are often nuanced and exist within the same domain, resulting in high inter-concept similarity. therefore, key requirements considered during the implementation are fast training, the capacity to learn from a few labeled samples, and the ability to distinguish between topically similar instances. the concept over time analysis workflow is internally structured as a three-step pipeline, encompassing: establishing the initial search space, fine-tuning the sentence transformer model, and computing the results. in the subsequent sections, we provide a detailed explanation of each step. the initial search space to initiate the concept over time analysis, users provide concepts along with proper descriptions. the descriptions are embedded with a sentence transformer model to generate an initial representation of each concept. the model is configurable during the tool setup and defaults to a pretrained multilingual clip @xcite following this, the concept representations are employed to identify semantically similar sentences, leveraging the existing semantic similarity search feature within dats. each uploaded document runs through a multi-step pre-processing pipeline, including sentence splitting and sentence embedding, with the results stored in weaviate foot_4 , an open-source vector database. consequently, the vector database is utilized to retrieve the top k similar sentences for each concept representation, along with their respective similarity scores. k is configurable in the ui and defaults to 1000. the set of returned sentences is considered as the search space, which remains constant in the subsequent steps. however, the ui allows resetting the search space and restarting the analysis process. the initial concept descriptions play a pivotal role in the entire analysis, determining the sentences considered during the analysis. this step is only executed once at the beginning, but crucial to the process as it limits the sentences to the concepts' domain and fastens the following steps. model fine-tuning users provide feedback in terms of labeling sentences with concepts. this step is skipped, if there are insufficient annotations for any given concept. the minimum labeled examples per concept can be configured in the ui, however, our experiments suggest that 4, 8, and 16 are good thresholds to start the fine-tuning. a contrastive training dataset is generated based on the users' annotations, which is then used to finetune a pre-trained sentence transformer model (more details in section 5). the model trains asynchronously in the background for 1 epoch, the frontend is informed about status changes and displays them accordingly. ## experiments few-shot fine-tuning of sentence transformers has been shown to achieve high accuracy with few labeled data by @xcite . our following experiments assess the suitability of this approach for the concept over time analysis extension, while also gaining insights into its limitations. we formulate the following requirements that our system should meet: require few labeled samples, enabling users to quickly achieve satisfactory results. ensure short training time to facilitate an interactive user experience with fast iterations. improve performance with more training samples rather than more training time, allowing iterative refinement of the model. ## conclusion in this paper, we presented the concept over time analysis extension, a machine-learning-based tool facilitating the identification and analysis of userdefined concepts within large datasets over time. the proposed tool empowers users to define concepts, identify relevant sentences, and visualize results in a timeline through a unified and interactive interface that is fully integrated within the discourse analysis tool suite. embracing the humanin-the-loop paradigm, the system iteratively refines the underlying model and enhances the timeline analysis with the help of user feedback. it employs few-shot sentence classification utilizing sentence transformer models for sentence and concept representations. with minimal human feedback, the model is fine-tuned on a contrastive dataset generated from the provided annotations. this results in a powerful tool for qualitative data analysis in large datasets. the chronological overview of custom concepts allows researchers to identify patterns, trends, and shifts in discourse over time. beyond discourse analysis, the extension supports various applications, including visualizing topics or sentiments over time. this work centered on analyzing and visualizing concepts in texts over time. in the next iteration, we plan to support image data, i.e., using crossmodal sentence transformers powered by clip, and adapting the interface for image processing. we also aim to fine-tune sentence transformers with adapters to further reduce training time while maintaining comparable performance.