What makes you change your mind? An empirical investigation in online group decision-making conversations (2022)

Abstract: People leverage group discussions to collaborate in order to solve complex tasks, e.g. in project meetings or hiring panels. By doing so, they engage in a variety of conversational strategies where they try to convince each other of the best approach and ultimately reach a decision. In this work, we investigate methods for detecting what makes someone change their mind. To this end, we leverage a recently introduced dataset containing group discussions of people collaborating to solve a task. To find out what makes someone change their mind, we incorporate various techniques such as neural text classification and language-agnostic change point detection. Evaluation of these methods shows that while the task is not trivial, the best way to approach it is using a language-aware model with learning-to-rank training. Finally, we examine the cues that the models develop as indicative of the cause of a change of mind.

## introduction research in group decision-making has shown that a group that collaborates in order to make a decision can outperform even the most knowledgeable individual within it @xcite . people engage in discussions in a variety of settings, such as project meetings and study groups. in these scenarios, people incorporate a variety of conversational strategies to introduce their ideas and convince each other of them, aiming ultimately to reach a consensus. fundamentally, before committing to a decision, most of the participants in a group have different ideas of what the correct answer might be, but through discussion they are able to convince each other, and ultimately some of the participants change their mind. while previous research has shown that people who reach a consensus tend to perform better at certain tasks @xcite @xcite , how people reach a consensus is understudied. successfully identifying what makes someone change their mind, is an important step in studying group dynamics, persuasion and collaboration. in this work, we take advantage of a dataset previously introduced by us @xcite , which contains group discussions of people solving a cognitive task. the dataset contains 500 dialogues, where people engage in various deliberation patterns to communicate their solution to the problem. the participants are presented with the wason card selection task @xcite , which is a classic problem used in the study of decision making and has been useful in testing the potential benefits and mechanisms of group discussion @xcite . the wason card selection task provides a controlled setup with quantifiable measures of success and improvement, which makes it very suitable for the study of individual biases and strategies. in the example in figure 1 , participants engage in a collaborative discussion where they iterate through 3 different solutions, where one of the participants changes their mind twice (in utterances 3 and 6). in the example, conversation utterances 2 and 5 are the arguments that cause that change of mind, and are the target utterances that we would like to predict. put formally, in order to investigate what made someone change their mind in group decision-making conversations we formalise the task as detecting the utterance that causes the change of mind (or conversational turning point, which is used in this paper interchangeably). in this work, we draw similarities between conversational turning points and change point detection. change point detection investigates when a change will occur in a stream of data, and is traditionally applied in domains such as finance @xcite , engineering @xcite , climate data @xcite , and genetics, @xcite . change point detection is concerned with either identifying a change post-hoc (offline change point detection or segmentation), or predicting a change point before it occurs -online change point detection @xcite . in this work, we are concerned with the latter -identifying a change of mind before it occurs. we are doing this by trying to predict which utterance will cause a change of mind. to evaluate our approaches, we develop a framework that quantifies the performance of models for change point detection in conversations, adopting practices from previous work @xcite . in terms of modelling, we first adapt a method for bayesian online change point detection @xcite , that was previously used in engineering and finance. this method is language-agnostic as it ignores any kind of linguistic cues. next, we explore standard approaches to text classification as a method for predicting conversational turning points, showing that they are comparable to the language-agnostic models. we further improve on these methods, by investigating learning-to-rank training for the prediction of what causes a change of mind. we demonstrate that by altering the training procedure and by incorporating the ranknet loss @xcite , we can substantially improve over the language-agnostic and text classification approaches. overall, our results demonstrate that the task of detecting conversational turning points is feasible but not trivial. approaches such as bag-of-words, or a simple all-positive baseline for change point detection have a performance of 0.18 area under the precision-recall curve. on the other hand, a combination of our learning-to-rank model and a positional prior led to an auc of 0.25. finally, we conclude this study with a qualitative analysis, where we demonstrate different patterns and linguistic phenomena that may indicate a cause of change of mind. ## related work the effect of conversation in group decisionmaking has previously been explored in the field of psychology. @xcite and @xcite show that there are conditions where a group of people who collaborate on a problem can outperform even the best individual group member. moreover, previous research @xcite has shown that groups of people who can reach a consensus through discussion have a higher group performance than just discussing or voting on a solution. @xcite has found that disagreement markers at the beginning of the conversation lead to productive discussions. likewise, both de kock and vlachos (2021) and hallsson and kappel (2020) study the effect that disagreement has on constructive conversations, showing how people who are disagreeing with each other can work together. therefore, we hypothesise that it is interesting to study the conversations where someone changes their mind i.e. they disagreed at first but ultimately reach a consensus. other research is concerned with which specific linguistic phenomena are associated with conversations that can change someone's mind (or persuasive conversations). @xcite investigate how topics and discourse change during a conversation, as well as their contribution to the persuasiveness of the conversation. similarly, @xcite analysed the prevalence of claims and premises in persuasive vs. non-persuasive dialogues. both of these papers leverage the online forum change my view, where participants argue pro and against a certain topic. unfortunately, the topics discussed in this forum are open to interpre-tation and personal opinion. therefore, they do not have a clear quantitative measure of whether someone changed their mind. further, while @xcite and @xcite study which phenomena may indicate that the conversation is more persuasive, they do not try to predict when will someone change their mind, and what is the specific utterance that caused that. identifying when a change in a sequence of observations occurs is traditionally studied in the context of change point detection, in the field of signal processing @xcite . formally defined, if we observe a sequence of a variable [x 1 , x 2 , ..., x n ], a change point occurs when two adjacent elements of that observation differ substantially. another way to define change points is by treating them as delimiters between different subsets of observed data. in this work, we adopt a version of the former definition -we are interested in the event that causes a subsequent observation to differ substantially from the previous ones. in terms of methods, change point detection can be broadly divided into online and offline methods. online methods (adams and mackay, 2007) focus on detecting a change point in a stream of data, and are evaluated based on the ability to predict a change point before it occurs (i.e. before the value changes substantially). on the other hand, offline @xcite methods by design work retrospectively on a sequence of datapoints, aiming at solving the task of segmentation. offline methods incorporate bi-directional information to determine when a change point occurred (i.e. the data points before and after the change point), whilst online methods rely only on the observed information. in this work we focus exclusively on online change point detection, as we would like to predict what causes a change before we observe the change it causes. arguably, detecting a change of mind post-hoc should be a more trivial endeavour, as s model could learn cues such as agreement markers or solution proposals. a different way to approach this would be from the point of view of survival analysis and reliability engineering @xcite @xcite . previous research relies on the concept of hazard function (also referred to as "time-to-failure") which is defined as the instantaneous risk of an event occurring at a point in time. the premise is that, as more time passes, the likelihood of an event occurring increases. practically speaking, in engineering the hazard function captures the intuition that as more time passes since the last maintenance, the likelihood of a breakdown of apparatus increases. we hypothesise that we observe a similar phenomenon in conversations -as the dialogue progresses, it is more likely for a participant to change their mind. ## data in this work, we are investigating what makes someone change their mind in group decision-making. in order to select a dataset to work on, we considered the following factors: statistics of the delidata corpus are presented in ## experimental setup using the gold and the weakly supervised sets introduced in section 3, we train all of our models with the following setup. all models are trained on the 220 dialogues from the weakly supervised set. the 50 gold annotated dialogues are split into test and validation sets, of 40 and 10 dialogues respectively. the validation sets are exclusively used for model selection of the text-based models. to train the linguistic and learning-to-rank models, we used a similar training setup. both models embed the input using the bart embedding layer @xcite . following, we added two fullyconnected layers with size of 1024 and 0.3 dropout between each of the layers. finally, we use a simple sigmoid function to perform the final classification. both models are trained using a batch size of 32 using the adadelta optimizer (zeiler, 2012) and were trained for 100 epochs, saving the iteration that has the best area under the precision-recall curve. ## results on we use 3 evaluation measures to compare the models -micro (utterance level) and macro (dialogue level) averaged area under the precisionrecall curve, and the precision-recall cutoff pointthe point where the precision and recall are equal. the reason to use these evaluation measures is three-fold. first, since change point detection typically deals with very imbalanced data, we need measures that are robust when the class of interest is under-represented. when dealing with heavily skewed data, @xcite argue that the area under the roc curve gives an overly optimistic estimate of the performance, and thus area under the precision-recall curve is a more appropriate measure. secondly, while evaluating our models, we noticed that different models have different precision and recall characteristics. for example, some of our models had very high precision, or very high recall, whilst producing comparable f-measures. in order to give a fairer comparison of the overall model performance, we report the micro-and macro-average area under the precisionrecall curve. finally, while area-under-the-curve gives a good estimation of performance, it doesn't give a lot of intuition of how the model will perform in terms of precision and recall when used in a practical setting. thus, we also report the precision-recall break-even point to show the relative predictive power of each model. in the best performing stand-alone model is achieved by training the linguistic model in a learning-to-rank setup (result 7), achieving micro and macro averaged auc of 0.23 and 0.26 respectively. further, we experimented with combining language-independent and language-agnostic models. in the context of this paper, we incorporated a simplistic combination -if either of the combined models predicts a conversational turning point we consider this as a positive signal. analysing the results, we observe that incorporating the sequence length and the bocp (adams and mackay, 2007) with all linguistic models can yield a substantial improvement. by combining the sequence length with the neural learning-to-rank model, we improve the performance to 0.25 micro auc and 0.30 p-r break-even point. in summary, while the neural models provide good stand-alone performance, they don't capture all of the information required for a prediction of a conversational turning point. namely, a substantial signal is carried by a positional information of where you are in a dialogue (captured by the sequence length probability), as well as patterns in how people discuss solutions (bayesian online change-point detection). ## qualitative study in order to gain some understanding of how each of the methods works, we qualitatively evaluate models' predictions. full conversation and model predictions are presented in appendix a. we use lime @xcite to find out which words are indicative for the positive predictions. we incorporated lime's to explain the prediction of the positive (cause of change of mind) class. the way method works is by first randomly perturbing features from the input, and then by learning linear models on the neighbourhood data to explain the label of interest. using this workflow, we extracted common words for each of the methods and for the rest of the section we present some of the findings. if we consider a pair of utterances that contain group interaction in the form of user mention: "utt1: <@xmath0: but then again most people get this wrong then it cant be as easy as we think surely". here both the bag-of-words baseline and the neural linguistic model classified the second utterance as a cause of change of mind. interestingly, the models gave weight to different features. the bag-of-words identified words such as "easy", "people" and "wrong" as important, which are part of an argument. on the other hand, the neural linguistic model put by far the highest weight on the participant mention, which is not related to the task at hand, but rather to the group dynamics. this observation is also supported by previous research @xcite , which argues that group dynamics play important role in collaboration. looking into the cases where one of the models predicted a cause of change of mind one utterance before the cause (as we allow for off by 1 errors), we consider the following pair of utterances: "then yeah we d have to make sure two vowels or two even numbers @xmath1. here the neural learning-to-rank model, predicted a cause of change of mind, and some of the words with the highest weight were "odd", "turn", and "need". we hypothesise that the model learned to recognise argument markers as suggestive for a cause of future change of mind. similarly, in the example "<@xmath2, <@xmath3. generally, the qualitative analysis shows that our best model (learning-to-rank) learnt to recognise argument cues as indicative of a conversational turning point. the model identified words that are related to the task such as card mentions or specific terms of the wason card selection task. that said, this could be a drawback -it is unclear how such models would perform for a different task, where the vocabulary is substantially different. ## conclusions in this work, we investigated methods for detecting the utterances that make someone change their mind, in the context of a recently introduced dataset containing group discussions of people collaborating to solve a task. we demonstrate that the best performance is achieved by combining a text-based model with a language-agnostic ones (such as positional information). in future work, we want to leverage the proposed approach to develop a system that can generate utterances that cause a change of mind in order to enhance group decision-making.