W ild V is: Open Source Visualizer for Million-Scale Chat Logs in the Wild (2024)

Abstract: The increasing availability of real-world conversation data offers exciting opportunities for researchers to study user-chatbot interactions. However, the sheer volume of this data makes manually examining individual conversations impractical. To overcome this challenge, we introduce WildVis, an interactive tool that enables fast, versatile, and large-scale conversation analysis. WildVis provides search and visualization capabilities in the text and embedding spaces based on a list of criteria. To manage million-scale datasets, we implemented optimizations including search index construction, embedding precomputation and compression, and caching to ensure responsive user interactions within seconds. We demonstrate WildVis’ utility through three case studies: facilitating chatbot misuse research, visualizing and comparing topic distributions across datasets, and characterizing user-specific conversation patterns. WildVis is open-source and designed to be extendable, supporting additional datasets and customized search and visualization functionalities.

## introduction while hundreds of millions of users interact with chatbots like chatgpt (malik, 2023), the conversation logs remain largely opaque for open research, limiting our understanding of user behavior and system performance. recently, initiatives such as wildchat (zhao et al., 2024) and lmsys-chat-1m (zheng et al., 2024) have released millions of real-world user-chatbot interactions, offering rich opportunities to study interaction dynamics. however, the volume and complexity of these datasets pose significant challenges for effective analysis. to help researchers uncover patterns and anomalies within these vast chat datasets, we introduce wildvis, an interactive tool for exploring millionscale chat logs. wildvis enables researchers to find conversations based on specific criteria, understand topic distributions, and explore semantically similar conversations, all while maintaining efficiency. figure 1 illustrates an example search using wildvis, applying criteria such as the keyword "election," conversations with more than two turns, and chats from users in florida, among others. wildvis features two main components: an exact, compositional filter-based retrieval system, which allows users to refine their search using ten predefined filters such as keywords, geographical location, ip address, and more. the second component is an embedding-based visualization module, which represents conversations as dots on a 2d plane, with similar conversations positioned closer together. both components are designed to scale to millions of conversations. a preliminary version of the tool, which supported filter-based retrieval for one million wildchat conversations, was accessed over 18,000 times by 962 unique ips in july and august 2024 alone. the latest release, described in this paper, extends support to both components for wildchat and lmsys-chat-1m. in this paper, we present the design and implementation of wildvis, discussing the strategies employed to scale to million-scale datasets while maintaining latency within seconds. we also showcase several use cases: facilitating chatbot misuse research (brigham et al., 2024; mireshghallah et al., 2024), visualizing and comparing topic distributions between wildchat and lmsys-chat-1m, and characterizing user-specific conversation patterns. for example, wildvis reveals distinct topic clusters such as midjourney prompt generation in wildchat and chemistry-related conversations in lmsys-chat-1m. additionally, we ob-serve that wildchat exhibits a generally more creative writing style compared to lmsys-chat-1m. as an open-source project, wildvis is available at github.com/da03/wildvisualizer under an mit license, and a working demo can be accessed at wildvisualizer.com. ## user interface wildvis consists of two primary pages-a filterbased search page and an embedding visualization page-along with a conversation details page. these pages are designed to provide users with both high-level overviews and detailed insights into individual conversations. ## system implementation wildvis is designed to efficiently process largescale conversational datasets. ## use cases this section presents several use cases that demonstrate the potential of wildvis. it is important to note that wildvis is designed primarily for exploratory data analysis rather than for final quantitative analysis. data wildvis currently supports two datasets: wildchat (zhao et al., 2024) and lmsys-chat-1m (zheng et al., 2024). these datasets are integrated into the system by building elasticsearch indices and precomputing the 2d coordinates of a randomly selected subset of conversations for embedding visualization. ## conclusion in this paper, we introduced wildvis, an interactive web-based tool designed for exploring largescale conversational datasets. by combining powerful search functionalities with intuitive visualization capabilities, wildvis enables researchers to uncover patterns and gain insights from vast collections of user-chatbot interactions. the system's scalability optimizations ensure efficient handling of million-scale datasets, while maintaining a responsive and user-friendly experience. wildvis fills a gap in existing tools by providing a specialized platform for visualizing and exploring chat datasets, which are inherently challenging to analyze using generic dataset viewers. our use cases demonstrate the tool's potential to replicate and extend existing research on chatbot misuse and user self-disclosure, as well as to facilitate topic-based conversation exploration.