Mitigating Topic Bias when Detecting Decisions in Dialogue (2021)

Abstract: This work revisits the task of detecting decision-related utterances in multi-party dialogue. We explore performance of a traditional approach and a deep learning-based approach based on transformer language models, with the latter providing modest improvements. We then analyze topic bias in the models using topic information obtained by manual annotation. Our finding is that when detecting some types of decisions in our data, models rely more on topic specific words that decisions are about rather than on words that more generally indicate decision making. We further explore this by removing topic information from the train data. We show that this resolves the bias issues to an extent and, surprisingly, sometimes even boosts performance.

## intro we spend a lot of our time in meetings. recordings of such meetings in the form of video or audio recordings or transcripts can be a valuable resource, but we need automatic processing and summarization methods if we are to be able to quickly search and retrieve the information we need. according to user surveys, the primary requirement of users from a meeting summarization system is a record of the decisions made @xcite . it can allow tracking of decisions and the reasoning behind them, as well as alternatives that were proposed and discussed. previous work on the task of automatic decision detection (e.g. @xcite @xcite shows that the problem is challenging: performance is limited @xcite unless strong assumptions about the nature of the data are made @xcite . e.g., assuming particular structure of the dialogue, rather than learning it from data. one reason for this is the lack of large datasets for the task. here, we show that previous models are also affected by another issue resulting from lack of data: topic bias. the intuition behind this problem is that the models might pick up on words that decisions are about instead of words that generally indicate decision making. as an example of this we provide the decision utterance -we agree to use a battery as a power source. a decision detection model might pick up on battery or power source as indicating decision making, simply because these phrases are something that often accompany decision in our data. however, a more unbiased model would ideally pick up on we agree as indicating decision making. our goal here is to explore and mitigate this problem by manually removing topic specific words, preventing the model from becoming topically biased. the contributions of this paper are two-fold. first, we present a deep learning based prediction model for a decision detection task. second, we give an analysis of topic bias in the data and models for this task, and show how our model can be made less susceptible to this bias compared to previous approaches. we make all our code and data publicly available. foot_0 2 related work some work in decision detection treats it as a text classification problem, and in some domains this is successful; @xcite achieve good accuracy detecting software architecture decisions in issue tracking systems. the same approach can be applied to face-to-face meeting dialogue, classifying individual utterances as decision-related or not on the basis of a range of lexical, structural and semantic features; but in this domain performance is lower @xcite . @xcite improve on this by considering the structure of the decision-making dialogue: they propose a set of decision-specific dialogue acts (ddas) and a model using support vector machines (svms) to classify each dda, using the outputs to predict decision discussion regions. similarly, @xcite explore real-time decision detection. further improvements have been shown via more explicit modeling of decision-making dialogue structure, encoded as probabilistic graphical models, and including non-lexical and prosodic features @xcite , but at the cost of assuming a fixed structure to a discussion rather than learning it from data. in contrast to related work, our primary focus is exploring the, thus far unaddressed, topic bias issues rather than maximum performance. consequently, we opt for simpler models that use only the text without additional features. we include one traditional and one deep learning based model. ## dataset we use the dataset introduced by @xcite , an annotated subset of transcripts from the ami meeting corpus @xcite covering 17 meetings in which actors stage a simulated meeting with the task of designing a remote control. each utterance is annotated with one or more of four specific decision dialogue acts (ddas): issue (i), resolution proposed (rp), resolution restated (rr), and agreement (a). categories rr and rp are both very low in number, which would likely hinder deep learning approaches. however, they are conceptually very similar, so we decided to merge them into a single category we denote as r. the annotations are multilabel (one utterance can perform more than one dda), although it is quite rare for an instance to have multiple labels (less than 1%). other available utterance metadata includes speaker id, timestamp, and a decision id (only for dda utterances). the total number of utterances in the dataset is 15680. ddas are rare, with each category making up only 1-2% of utterances. the sparsity of the decision acts presents a considerable problem for all work on this data set. ## methodology as part of our methodology, we next describe the models and evaluative metrics we employ. ## experiments and results experiment setup we evaluate the models using leave one out cross-validation. in each iteration, we train the models on 16 meetings and test them on the remaining meeting. for both the ule and seg evaluation setups, scores are calculated at the level of the meeting and averaged. no masking with masking baseline bert baseline bert p r f1 p r f1 p r f1 p r f1 i .152 .440 .209 .221 .314 .232 .140 .530 .210 .237 .361 .263 r .210 .713 .304 .236 .490 .292 .174 .769 .271 .294 .527 .333 a .175 .845 .283 .257 .658 .352 .165 .844 .270 .255 .627 .343 seg .337 .885 .527 .419 .761 .540 .355 .906 .510 .427 .770 .547 for the bert-lstm model we use the adam (kingma and ba, 2015) optimizer with learning rate 10 -4 and minibatch size 32. out of the 16 training meetings we set one aside as a development set for early stopping. we train the model until there has been no improvement in score for any of the evaluation setups on the development data for 5 consecutive epochs. furthermore, we found that due to the small data set size, this training regime sometimes produces very bad models (depending on random initialisation). we circumvent this by training it several (in our case 16) times with different development meetings and different random initialisations. we use on the test set the variant that has highest test set confidence scores. 7 6 making the baseline stronger than in a realistic scenario. 7 this in no way uses the test set labels. ## conclusion we have explored the problem of topic bias in detecting decision dialogue acts (ddas). in particular, we have identified bias for the issue and resolution types of ddas. we experimented with mitigating the bias by manually identifying and removing topic related words and our main finding is that, while this partially mitigates the bias issues and sometimes even improves performance. however, to further confirm these findings more experiments on other, larger data data sets are required. there are several avenues of future work. these include exploring models that capture speakers, using non-decision dialogue acts as additional information, or pretraining language models on decision-related sentences. the immediate direc-tion, however, is to increase the size of dda annotated data and include a more diverse set of topics.