Disruptive Talk Detection in Multi-Party Dialogue within Collaborative Learning Environments with a Regularized User-Aware Network (2022)

Abstract: Accurate detection and appropriate handling of disruptive talk in multi-party dialogue is essential for users to achieve shared goals. In collaborative game-based learning environments, detecting and attending to disruptive talk holds significant potential since it can cause distraction and produce negative learning experiences for students. We present a novel attention-based user-aware neural architecture for disruptive talk detection that uses a sequence dropout-based regularization mechanism. The disruptive talk detection models are evaluated with multi-party dialogue collected from 72 middle school students who interacted with a collaborative game-based learning environment. Our proposed disruptive talk detection model significantly outperforms competitive baseline approaches and shows significant potential for helping to support effective collaborative learning experiences.

## introduction automatic analysis of dyadic dialogue utilizes a broad range of methods for intent recognition @xcite @xcite . compared to dyadic conversations, multi-party conversations are characterized by a high degree of complexity due to multi-way group interactions, thus, multi-party dialogue models should take into account group dynamics to reliably model phenomena. for example, previous research investigated giving less weight to participants whose convergence behaviors differ from the rest of the group @xcite to examine which utterances should be clustered together (i.e., conversation threads) in multi-party dialogues @xcite . in education, computer-supported collaborative learning environments promote social aspects of learning through the use of a variety of technological and constructive pedagogical strategies, including problem-based learning and inquiry learning @xcite @xcite . collaborative game-based learning environments often provide students with in-game chat features to help promote open discussion and negotiation among team members, facilitating the coordination of their in-game learning activities @xcite . however, students are not always effective collaborators and may engage in improper communicative behavior, distracting from the group learning experience. the presence of negative socio-emotional engagement in collaborative learning environments can result in disruptive talk and can function as a barrier to the development of high-quality collaborative communication. previous work on detecting talk that can cause negative socio-emotional engagement (e.g., offtask behavior, bullying, disruptive talk) in collaborative learning environments investigated computational approaches using language models ranging from classic approaches (e.g., n-grams) and word embedding approaches (e.g., bert). these language models have been combined with classic techniques (e.g., logistic regression, random forest) and deep learning techniques (e.g., longshort term memory networks) @xcite @xcite . however, the previous work either makes utterance-by-utterance predictions without taking disruptive talk detection in multi-party dialogue within collaborative learning environments with a regularized user-aware network context into account or treats the entire multi-party conversation sequence as a continuous dialogue flow, despite the potential presence of multiple concurrent message threads with different topics. in this paper, we propose a novel attentionbased, regularized user-aware modeling approach for detecting disruptive talk in multi-party dialogue within a collaborative game-based learning environment. we investigate the use of target-user embeddings to help the prediction model determine the disruptiveness of the sequence more accurately with an additional user-specific network and attention mechanism. we also investigate a sequence-level dropout mechanism during training as a regularization technique that could help avoid overfitting possible diluted conversation sequences (i.e., presence of multiple threads in a sequence) in training data. experimental results demonstrate that our attention-based, regularized user-aware model offers great potential for addressing disruptive talk detection in multi-party dialogues. ## related work diverse prediction tasks have analyzed multi-party dialogue focusing on the asynchronous and entangled nature of group conversations, such as dialogue act classification using group thread history, and thread detection as well as cyberbullying and toxic message detection within group conversations @xcite @xcite @xcite . @xcite investigated classic machine learning approaches for dialogue act classification, such as naïve bayes, support vector machines, and conditional random fields, along with contextual, structural, keyword, and dialogue interactionbased features of utterances for dialogue act classification in multi-party live chat datasets. as a sub-task of a disaster response mission knowledge extraction task, @xcite proposed a deep learning-based divide&merge architecture utilizing lstm and cnn for predicting dialogue acts. @xcite investigated the use of dialogue act prediction utilizing conditional random fields and elmo contextualized word embeddings in multiparty team communication for providing adaptive team training support. as multiple participants are involved in multiparty conversation, disentanglement of the conversation based on relevancy is another important task, which could enhance the conversational relevance rate of automated dialogue agents @xcite or improve summarization quality @xcite . @xcite proposed three lstmbased context-aware thread detection architectures that automatically captures conversation threads in multi-party and multi-thread conversations, where the proposed model predicts which existing thread the current utterance belongs to (or whether it creates a new thread). another task that has received considerable attention in multi-party conversation is cyberbullying. the ability to detect bullying or toxic behavior is crucial to protecting users from cyberbullying. in particular, researchers are increasingly interested in toxic behavior in multiplayer games, such as multiplayer online battle arena (moba) games, where players compete against other teams in virtual online game environments @xcite . @xcite used random forest classifiers to detect toxic behavior in league of legends using in-game performance, user reports, and chat data. the conversation data included 590,000 utterances, which were labeled via crowdsourcing on whether the conversation was toxic or not. @xcite presented a linguistic analysis of genderbased toxic language usage in a dota 2 chat dataset and investigated naïve bayes classifiers with three different laplace smoothing parameters as an automatic approach for sexist toxic comment detection. a significant presence of gender discrimination in online games, mainly by young males and intense players, was revealed in their qualitative analysis. students' conversations can create disruption in collaborative learning environments, impeding collaborative learning processes. recent research on bullying, off-task behavior, and disruptive talk in collaborative learning environments examined a range of word embedding techniques as well as a variety of classical machine learning and deep learning techniques @xcite @xcite . @xcite explored the automatic detection of aggressive behavior (i.e., bullying) in two k-12 computer-supported collaborative learning environments. they used unigrams to represent words and examined machine learning approaches such as naïve bayes with laplace smoothing, decision tree classifiers, and feedforward neural networks. the prediction results suggest that approaches based on deep learning outperform other classical machine learning approaches. @xcite used dialogue analysis to identify if students' messages were on-task or offtask during collaborative game-based learning. to develop a model capable of reliably detecting offtask behavior, they investigated three different word embedding approaches (i.e., word2vec, elmo, and bert), various history lengths of previous utterances, and two deep learning and classical machine learning classifiers were trained on a feature set containing contextual information extracted from student chat messages. the empirical evaluations indicated that the lstmbased off-task behavior detection model with bert embeddings outperformed other baseline approaches. @xcite presented an lstmbased disruptive talk detection framework in a multi-party dialogue dataset from a collaborative game-based learning environment, utilizing features from chat messages, a range of linguistic features, gender, and pre-test scores. while this work has the potential to improve learning experiences by detecting disruptions within collaborative learning settings, they disregard the unique characteristics of multi-party dialogues. in our work, we improve predictive performance of disruptive talk detection models by incorporating an additional network that embeds the characteristics of the user of a target utterance and a sequence-level dropout mechanism. ## corpus we next describe the collaborative game-based learning environment and its chat-interface, dataset collected from two field studies, and disruptive talk annotation process. ## result and discussion the performance enhancement with the sequence dropout training mechanism suggests that the conversation sequences may have contained noise due to the presence of multiple conversation threads, and that the model had some trouble determining how to extract the essential parts of the conversation sequences that could help with disruptive talk predictions. the model was given the opportunity to learn multiple variants of utterances from the same sequence because of the random dropping of a different subset of sequences at each training epoch. it is possible that this method could help achieve improved predictive performance by regularizing the disruptive talk detection models to effectively deal with noisy conversation data. lastly, we note potential limitations of our research. because of the nature of stratified grouplevel sampling where the sampling procedure must take into account both the label distribution and the group index, it is not possible to apply the exact same distribution across different folds, which could result in large performance variations between folds. in addition, while our proposed modeling approach demonstrated a promising result in our testbed collaborative game-based learning environment, the proposed model could be evaluated with other computer-supported collaborative learning environments to demonstrate generalizability of the technique. ## conclusion multi-party dialogue modeling poses significant challenges because of the complexity driven by group dynamics characterized in multi-party conversations. detecting disruptive talk in collaborative game-based learning environments is crucial to support high-quality collaborative learning. we have presented a novel deep learningbased disruptive talk detection model that incorporates a user-aware attention network and a random sequence dropout training mechanism, where the model utilizing both approaches significantly outperform the baseline approaches. the proposed model shows significant promise for addressing key challenges in multi-party dialogue prediction. in the future, it will be important to test our model's capability with multi-party dialogue corpora from other computer-supported collaborative learning environments to test the generalizability of our model. it will also be important to implement the disruptive talk detection model in a real-time setting and investigate how it informs adaptive support for collaborative student learning.