Improving the Generalizability of Collaborative Dialogue Analysis With Multi-Feature Embeddings (2023)

Abstract: Conflict prediction in communication is integral to the design of virtual agents that support successful teamwork by providing timely assistance. The aim of our research is to analyze discourse to predict collaboration success. Unfortunately, resource scarcity is a problem that teamwork researchers commonly face since it is hard to gather a large number of training examples. To alleviate this problem, this paper introduces a multi-feature embedding (MFeEmb) that improves the generalizability of conflict prediction models trained on dialogue sequences. MFeEmb leverages textual, structural, and semantic information from the dialogues by incorporating lexical, dialogue acts, and sentiment features. The use of dialogue acts and sentiment features reduces performance loss from natural distribution shifts caused mainly by changes in vocabulary. This paper demonstrates the performance of MFeEmb on domain adaptation problems in which the model is trained on discourse from one task domain and applied to predict team performance in a different domain. The generalizability of MFeEmb is quantified using the similarity measure proposed by Bontonou et al. (2021). Our results show that MFeEmb serves as an excellent domain-agnostic representation for meta-pretraining a few-shot model on collaborative multiparty dialogues.

## introduction for many natural language processing applications, the ability to learn features that generalize well across multiple datasets is a key desideratum @xcite . this paper introduces a new multi-feature embedding, mfeemb, that increases the generalizability of models learned from collaborative multiparty dialogues. dialogues are different from single-author documents in that, along with textual information, they contain communication patterns that may serve as indicators of social dynamics. treating a dialogue as a mere text collection ignores valuable information. we advocate exploiting implicit features present in multiparty dialogues that are less vulnerable to distribution shifts resulting from task domain changes. this paper demonstrates the usage of mfeemb on a communication analysis task: conflict prediction. teamwork research faces a challenge of resource scarcity since the human subjects datasets are quite small (less than 100 samples), due to the difficulty of recruiting teams and the time consuming nature of many group tasks. a variety of social phenomena have been investigated within team communication research, including entrainment @xcite and emergent leadership @xcite . frequency of communication is not in itself a good predictor of team performance, but a meta-analysis conducted by @xcite that drew upon data from 150 studies conducted on 9702 teams concluded that high quality communication is positively related to team performance in many task domains. conversely, process conflict, "disagreement among group members about the content of the tasks being performed, including differences in viewpoints, ideas, and opinions" @xcite , is usually negatively correlated with taskwork success. our aim is to be able to learn a model to classify process conflict from multiparty dialogues that generalizes well across multiple tasks. we treat the task of conflict prediction as a binary classification task with high conflict and low conflict being the two classes; the ground truth used by the conflict prediction model is measured using a post-task team process conflict survey. this paper focuses on three collaborative problem-solving tasks: software engineering, search and rescue, and cooperative gameplay. our proposed embedding, mfeemb, leverages textual, structural, and semantic information from the dialogues by incorporating vocabulary, dialogue acts, and sentiment features. lexical embed-dings such as word2vec and bert @xcite show good performance across multiple nlp tasks on in-domain test sets but are less robust to domain shift. previous work identified that dialogue acts and sentiment sequences are informative features that predict conflict reliably even at the earliest stage of team problem-solving @xcite ; however, classifiers constructed using these features still experience lackluster transfer performance when applied to new datasets, particularly when detecting high conflict examples @xcite . to address this transfer problem, we propose the usage of mfeemb, specifically as a metapretraining representation to be used within a fewshot model. mfeemb combines the strengths of both domain-invariant and domain-specific features. this paper compares the generalizability potential of the mfeemb embedding vs. standard word embeddings using inter-class and intra-class based similarity measures, proposed by @xcite . then we evaluate the performance of mfeemb in a domain adaptation scenario in which the model is trained on discourse from one task domain and used to predict conflict in a different domain. our results show that: ## related work previous studies on group interaction tasks such as conflict prediction @xcite , disruptive talk detection @xcite , group satisfaction @xcite , and task performance prediction @xcite have focused on simply improving performance on in-domain datasets. very little attention has been paid to the problem of creating generalizable models for multiparty dialogue that can be used when training data is scarce. the intelligent tutoring system community has empirically assessed the generalizability of common natural language representations, such as bert and linguistic inquiry word count (liwc), across collab-orative problem solving tasks @xcite , but without investigating methods to improve generalizability. in domain adaptation, the goal is to train a model on data from a source domain that performs well on a test dataset drawn from a different target distribution. common nlp tasks (e.g., part-ofspeech (pos) tagging and named entity recognition (ner)) have been tackled using techniques including instance weighting @xcite or explicitly identifying feature correspondences between the domains @xcite ). an alternate approach is to learn a single representation that generalizes well across multiple domains. this can be done using few-shot learning @xcite , one of the most widely used approaches to dealing with resource scarcity. the traditional framework comprises meta-training and meta-testing phases, where the aim of meta-training is to learn universal representations from multiple domains. @xcite introduced a method that improves few-shot generalizability by making use of multiple datasets in order to learn a universal template. @xcite proposed selecting from universal representations (sur), which involves learning a multi-domain representation by training multiple feature extractors. a multidomain feature bank is used to select the most relevant feature during the learning phase. rather than seeking to learn the new representation entirely from data, our research exploits similarities in dialogue act sequences and sentiment patterns commonly observed during successful collaborative problem-solving. representation choice has been shown to place an upper bound on target domain performance @xcite . few-shot frameworks such as meta-pretraining then meta-learning (mtm) @xcite have assumed that word embeddings like bert that are trained on large datasets are the best choice for task agnostic pre-training. @xcite introduced a method to quantify the generalizability of a few-shot classifier under supervised, unsupervised and semi-supervised settings. this paper uses their inter-class and intra-class based generalizability measure to evaluate mfeemb vs. simple word-based embeddings under supervised classification scenarios. our research demonstrates that mfeemb is superior to word embeddings as a metapretraining representation. ## methodology this section describes our approach to learning a generalizable embedding from multi-party dialogues for conflict prediction. we discuss our datasets, introduce our embedding, and show how our technique can be used in combination with data augmentation and few-shot learning. ## conclusion and future work this paper introduces a multi-feature embedding (mfeemb) to improve the generalizability of multiparty dialogue models under resource scarcity scenarios. we propose the use of a combination of textual (words), structural (das), and semantic (sentiment, das) embeddings to reduce the performance loss due to natural distribution shift. experiments show that the multi-feature embedding performs significantly better than sentence (bert), dialogue act-only, sentiment-only, and word embeddings. our results demonstrate that mfeemb is a superior representation for meta-pretraining a few-shot model that works well across different collaborative problem-solving domains. our proposed data augmentation strategy successfully resolved the domain shift problem caused by task-specific vocabulary without perturbing the dialogue act and sentiment features. experiments with synthetic datasets show that synonym replacement with vocabulary drawn from a collaborative task outperforms generic synonym replacement with wordnet. it improves both the transfer accuracy and the test accuracy on the in-domain test set. note that we did not fine-tune the models on the target datasets, i.e., github and asist, and strictly re-port the model learned on the teams dataset. only the vocabulary of these datasets was used to boost the performance; explicit fine-tuning of the machine learning models could further improve the results.