Identifying Collaborative Conversations using Latent Discourse Behaviors (2020)

Abstract: In this work, we study collaborative online conversations. Such conversations are rich in content, constructive and motivated by a shared goal. Automatically identifying such conversations requires modeling complex discourse behaviors, which characterize the flow of information, sentiment and community structure within discussions. To help capture these behaviors, we define a hybrid relational model in which relevant discourse behaviors are formulated as discrete latent variables and scored using neural networks. These variables provide the information needed for predicting the overall collaborative characterization of the entire conversational thread. We show that adding inductive bias in the form of latent variables results in performance improvement, while providing a natural way to explain the decision.

## introduction online conversations are rampant on social media channels, news forums, course websites and various other discussion websites consisting of diverse groups of participants. while most efforts have been directed towards identifying and filtering negative and abusive content @xcite @xcite , in this paper we focus on characterizing and automatically identifying the positive aspects of online conversations @xcite @xcite . we specifically focus on collaborative conversations, which help achieve a shared goal such as gaining new insights about the discussion topic like response informativeness, engagement etc. rather than looking at the outcomes of such conversations (e.g., task completion @xcite )), we analyze conversational behaviors, specifically looking at indications of collaborative behavior that is conducive to group learning and problem-solving. these include purposeful interactions centered around a specific topic, as well as open and respectful exchanges that encourage participants to elaborate on previous ideas. to help clarify these concepts, consider the following conversation snippet. user a : we should invest in more resources to encourage young people to be responsible citizens. response option 1 : i wonder if more initiatives at grassroots level can help them to identify and understand issues of their local community more deeply. response option 2 : good point, i agree. we compare the two possible responses to user a's post. option 1 offers a balanced contribution, developing the idea presented in the original post and allowing the conversation to proceed. option 2, while polite and positive, is not collaborative as the initial idea is not expanded on. in fact, agreement is often used as a polite way to end conversations without contributing additional content. despite the positive sentiment, capturing the absence of balanced content contribution and the absence of idea development as different discourse behaviors, one can infer that it is not a collaborative conversation. while humans could tell the two apart by detecting constructive discourse behaviors, automatically capturing these behaviors is highly challenging. anecdotal evidence, collected by extracting features from conversation transcripts, can lead to conflicting information, as identifying collaborative behavior relies on complex interactions between posts. our main intuition in this paper is that reasoning and enforcing consistency over these behaviors can help capture the conversational dynamics and lead to more accurate predictions. our technical approach follows this intuition. we design a hybrid relational model that combines neural networks and declarative inference to capture high-level discourse behaviors. since we only have access to the raw conversational text, we model these behaviors as discrete latent variables, used to support and justify the final decision -whether the conversation is collaborative or not. explicitly modeling discourse behaviors as latent variables allows us to add inductive bias, constraining the representation learned by the neural model. it also provides a natural way to "debug" the learning process, by evaluating the latent variables activation. our experiments show that the joint model involving global learning of different latent discourse behaviors improves performance. we use the yahoo news annotated comments corpus @xcite , and expanded the annotation for the collaborative task. 1 2 task definition collaborative conversations are purposeful interactions, often revolving around a desired outcome, in which interlocutors build on each others' ideas to help move the discussion forward. collaborative conversations are an important tool in collaborative problem solving @xcite and require collaboration skills @xcite . we focus on identifying indicators of successful collaboration. we build on the work of @xcite , who released a dataset annotated for engaging, respectful and informative conversations, and annotate it for collaborative conversations, in which participants build on each other's words, provide constructive critique, elaborate on suggested ideas, generalizing them and synthesizing new ideas and knowledge in the process. during the annotation process, we identified several repeating behaviors (detailed below) that helped characterize and separate between collaborative and non-collaborative conversations. annotation process two annotators labeled the conversations based on these guidelines, with an accuracy in inter-annotator agreement of 81%. ## modeling collaborative behaviors identifying collaborative conversations requires characterizing nuanced behaviors. in previous work, this analysis was defined by extracting social and discourse features directly from the raw data. in contrast, we view this decision as a probabilistic reasoning process over the relevant conversational behaviors that were identified during the annotation process (sec. 2.1 and 2.2). since these behaviors are not directly observed, and have to be inferred from the raw conversational features, we treat them as discrete latent variables which are assigned together-with, and consistent-with, the final classification task. each behavior is captured by a binary latent variable, denoted as @xmath0, ..., h k , indicating if it's active or not in the given thread. these decisions are then connected with the final prediction, denoted y, a binary output value. this results in a factor graph (figure 1 ). each individual decision is scored by a neural net, and uses a set of features capturing relevant properties in the input conversation. to learn this model, we extend drail @xcite , a recently introduced framework for combining declarative inference with neural networks, described briefly in the following section. our extension allows for the introduction of discrete latent predicates into the model.