Team Conversational AI : Introducing Effervesce (2025)

Abstract: Group conversational AI, especially within digital workspaces, could potentially play a crucial role in enhancing organizational communication. This paper introduces Effervesce, a Large Language Model (LLM) powered group conversational bot integrated into a multi-user Slack environment. Unlike conventional conversational AI applications that are designed for one-to-one interactions, our bot addresses the challenges of facilitating multi-actor conversations. We first evaluated multiple open-source LLMs on a dataset of 1.6k group conversation messages. We then fine-tuned the best performing model using a Parameter Efficient Fine-Tuning technique to better align Effervesce with multi-actor conversation settings. Evaluation through workshops with 40 participants indicates positive impacts on communication dynamics, although areas for further improvement were identified. Our findings highlight the potential of Effervesce in enhancing group communication, with future work aimed at refining the bot’s capabilities based on user feedback.

## introduction in the era of digital workspaces, organizations are increasingly communicating using different online tools that facilitate interaction and collaboration on the level of the entire organization or in teams @xcite . recent studies have highlighted the importance of online collaboration software (ocss), particularly for teamwork, and even more specifically, for distributed, virtual teams @xcite @xcite . organizational virtual teams are relatively small, task-oriented groups of individuals who are often physically distributed to multiple locations nation-or worldwide, and mostly work technology-mediated toward a common goal @xcite . when shared physical premises are lacking, the importance of online collaboration software becomes even more evident: it becomes the site where both work-related and relational team processes take place (e.g., @xcite @xcite . ocss, thus, facilitate various team processes across temporal and physical boundaries, as well as allow team members to get to know each other by providing a shared platform for the team to socialize on @xcite . researchers have extensively discussed how technology integrates into organizational life: it shapes social action of organization members and technology itself is also shaped through people using it @xcite . during the past few years, the development of large language models (llms) and chatbots built using them has radically changed the type of technologies used in organizational communication. through the advances of communicative ai, the role of technology develops from a tool that affords communication to a tool that participates in human interaction. in communication scholarship, the term "communicative ai" has been coined to refer to devices, applications, and algorithms capable of communicating in natural language and adapting to real-life conversational situations (guzman and @xcite . in computer science, these applications have been discussed under the term conversational ai (e.g., @xcite . the future projections of companies such as openai even suggest that nonhuman conversational agents could soon be indistinguishable from humans @xcite . in this study, we start from the premise that communicative ai applications, communication tools that are enhanced with llms and generative ai (genai), could play a critical role in facilitating effective group conversations. traditional conversational ai applications are predominantly designed for one-to-one interactions in the form of chat @xcite , which also applies to the most widely used conver-sational ai tools such as chatgpt or microsoft copilot also used in a professional context. to facilitate team conversation and collaboration, the conversational ai should be able to take part in group conversations. this generates a need for models and applications that can support manyto-many conversations. such an ai application could enter the team ocs with its own account and join the conversation almost as a team member. in addition, it should be able to read the flow of conversation and adapt to the language style of the team. in this work, we introduce effervesce, an llmpowered group conversational bot operating on slack designed to integrate into group conversations and engage as an ai team member in the organization's digital workspace. to power our chatbot, or more accurately a socialbot @xcite , we evaluate various open source models that provide us with robust version control and help address data privacy concerns. increasingly, alternative open source llms are being introduced in multiple recent works, including llama @xcite , mistral-7b @xcite . we created a group-conversational dataset from the 1,608 messages posted on a slack channel of a single team. in our preliminary evaluations of various open-sourced llms with our group conversational dataset, we observed that such models struggled to capture the language style and structure of the conversational context. we selected the best-performing model, a fine-tuned variant of mistral-7b, to power effervesce. we addressed the identified issues with context understanding by fine-tuning the selected llm. we acknowledge that there are substantial costs and environmental implications associated with training and fine-tuning such large machine learning models @xcite . to minimize these effects, we experimented with a parameter-efficient fine-tuning (peft) technique, known as qlora @xcite . this method allowed us to update only a small fraction of the total 7+ billion parameters while maintaining the pre-trained model's performance. our fine-tuned version of the model managed to learn from the training data while maintaining a good generalization level. to assess the fine-tuned effervesce, we conducted a qualitative evaluation through 10 workshop sessions, involving 40 participants in total. the feedback was useful to guide future improvement in our approach and, among others, indicated that while the bot demonstrated improved conversation and context awareness, it responded too quickly and provided long detailed responses. the contributions of this work are as follows. in the following section, we summarize existing research in group conversational ai systems, technicalities and costs concerning the pre-training and fine-tuning of llms. in section 3, we discuss the methodology of this work, presenting details on our dataset, llm evaluation, and the fine-tuning approach that we employ. we describe the experiment and disseminate the results in section 4, while in section 5 we provide a discussion of the results and conclude this work. lastly, in section 6 we list the future work leads that emerge from this research. ## background and related work communication in organizations has increasingly shifted to online collaboration software, where teams collaborate in shared systems. nowadays, human users on such systems are increasingly accompanied by different ai tools designed to help their workflows, knowledge management, and communication. in general, the introduction of genai tools in work life is expected to shape agency and action in knowledge work: routines, processes, and also professional interactions @xcite previous studies on conversational bots in organizations show how ai agents can mediate human interaction and facilitate knowledge sharing @xcite @xcite . only a few studies have focused on bots that take part in group conversations @xcite , but these bots have represented pre-genai era bots with quite simple communication capabilities. however, most conversational ai systems used and studied so far have been applications that enable one-to-one or user-assistant interactions @xcite @xcite . consequently, research has focused on communication processes such as simple question-answering or knowledge sharing, without exploring the application of llms in realtime group conversations. more recent works analyze the value of contextual understanding in group conversation settings, particularly relevant in online digital platforms like community question and answering, slack, and reddit @xcite , where multiple members can engage in conversations across different channels, threads, and topics. various technical and design challenges arise when employing such multi-user conversational ai systems. most notably, the conversation ai system should be able to follow the structure of the conversation and take into account that there are multiple participants involved. these challenges require ai models that keep track of dynamic conversations, recognize multiple speakers, and follow the discussion's context. transformer-based architectures @xcite proved that contextual embeddings can capture special language features from text giving shape to the natural language processing (nlp) research landscape. this attribute has enabled the development of large language models (llm), which have shown superior performance on a wide range of benchmark tasks. early works like bidirectional encoder representations from transformers (bert) @xcite proved the power of pre-training deep transformer-based models on massive textual datasets to extract improved features from written language. bert was followed by other models like openai's generative pre-trained transformer (gpt) models @xcite , which demonstrated superior few-shot learning capabilities. more recent models such as google's language model for dialogue applications (lamda) @xcite , gemini @xcite , and deepseek-r1 (deepseek-ai, 2025) have become foundational models for various products and applications, including intelligent chatbots. these models often contain billions or even trillions of parameters, posing significant challenges for implementation. training and deploying such large models requires vast amounts of computational resources and energy, making them expensive and less accessible. fine-tuning these models for specific tasks can also be computationally intensive @xcite . recent works have introduced various parameter-efficient fine-tuning (peft) techniques as solutions to address these challenges. methods like low-rank adaption (lora) @xcite and quantized lora (qlora) @xcite provide alternative efficient techniques to fine-tune pre-trained llms for specific data and application contexts, by enabling training only on a small fraction of the model's parameters. the challenges of enabling a chatbot to adopt different roles in multi-user conversations have been identified and explored also by @xcite , who introduced an augmented and fine-tuned gpt-2 model @xcite , which emulates the persona of a target actor based on previous conversations they engaged with. their large-scale reddit dataset of 10.3 million conversations enabled fine-tuning without employing parameter-efficient techniques. however, such an approach can be expensive or not feasible, especially for smaller organizations or limited datasets. ## methodology in this section, we describe our approach to building and evaluating effervesce, our slack-based group conversation bot. first, we describe how we constructed the group conversation dataset from real slack messages. next, we explain the process of evaluating, selecting, and efficiently finetuning open-source llms, to power our chatbot. finally, we describe how we evaluated the bot's performance based on the quantitative metrics and the qualitative feedback we received from human users who interacted with our bot in workshop settings. ## experiment and results we present our experiments and findings on evaluating a set of llms on group conversational data and fine-tuning and evaluating an llm to power our group chatbot. first, we describe the experimental setup. then, we organize our results in two groups: 1) evaluating different llms with our group conversational data, and 2) fine-tuning and qualitative assessment of effervesce. ## discussion and conclusion in this work, we explored how effervesce, our group conversational chatbot, integrated with slack and designed to engage in real-time multi-actor conversations. we evaluated multiple open-source llms, fine-tuned mistral-7b-instruct-v0.1 model using the qlora technique, and evaluated effervesce's performance through quantitative metrics and qualitative user feedback. pre-trained models achieved lower perplexity scores, compared to their fine-tuned counterparts, when evaluated in our group conversational dataset. however, these foundation models performed worse based on bleu scores, suggesting their lack of alignment with the language style in the group conversation. fine-tuned models improved bleu scores consistently, but performed worse on the perplexity metric. given these findings, we selected mistral-7b-instruct-v0.1 model for further fine-tuning and powering effervesce due to its better performance in instruction-following, group context understanding, and higher response quality perceived by humans. perplexity decreased during the fine-tuning process, indicating that the model managed to learn the language style and patterns from our specific application data context. we evaluated the fine-tuned model on our test set, and it achieved an average perplexity score of 7.8, indicating the model did not overfit. through our qualitative evaluation, we received feedback regarding effervesce's performance in group conversation settings. the bot was perceived as too active during the first interactions, disrupting the flow of the conversation. we introduced a response probability parameter in the system, which helped to improve this concern for the following workshops. some users found effervesce's responses too long and overwhelming. we received feedback indicating the bot's language tone was found to be too friendly, using a lot of emojis, and informal language considering the professional context of evaluation. our chatbot also made mistakes when referring to users participating in the conversation. mistakes were in the form of typos and complete misses. some users felt the bot did not provide critical feedback when asked to facilitate their brainstorming session. effervesce demonstrates the potential of llmpowered multi-actor chatbots in digital workspaces to enhance group communication dynamics in organizations. fine-tuning improved its performance and alignment with the group conversation structure and dynamics. nevertheless, the user feedback pointed out further challenges that the bot faces. addressing the identified issues is crucial for further investigating how to make group conversational ai more effective. our work contributes to the growing research field of llm-powered multi-actor group conversation chatbots through the insights we provided regarding the llm fine-tuning, and practical integration and deployment process.