“ I ’ll be there for you”: The One with Understanding Indirect Answers (2021)

Abstract: Indirect answers are replies to polar questions without the direct use of word cues such as ‘yes’ and ‘no’. Humans are very good at understanding indirect answers, such as ‘I gotta go home sometime’, when asked ‘You wanna crash on the couch?’. Understanding indirect answers is a challenging problem for dialogue systems. In this paper, we introduce a new English corpus to study the problem of understanding indirect answers. Instead of crowd-sourcing both polar questions and answers, we collect questions and indirect answers from transcripts of a prominent TV series and manually annotate them for answer type. The resulting dataset contains 5,930 question-answer pairs. We release both aggregated and raw human annotations. We present a set of experiments in which we evaluate Convolutional Neural Networks (CNNs) for this task, including a cross-dataset evaluation and experiments with learning from disagreements in annotation. Our results show that the task of interpreting indirect answers remains challenging, yet we obtain encouraging improvements when explicitly modeling human disagreement.

## introduction humans are very good at interpreting indirect answers to polar questions. in conversations, even if direct answers are possible, humans often prefer indirect answers due to cooperativeness and to advance the dialogue @xcite . for dialogue systems and natural language processing (nlp) more generally, however, interpreting indirect answers remains a challenge @xcite . recent seminal work introduced circa, a new large-scale dataset containing pairs of polar questions and indirect answers in english @xcite . this allows for data-driven experiments in this question-answering domain. understanding indirect answers is a pragmatic problem, and even though humans typically have little difficulty in interpreting indirect answers, they may not all agree on a possible interpretation. work on learning from human disagreement has shown that incorporating disagreement from human annotation is not only noise, but can provide valuable information @xcite @xcite . motivated by these two lines of research, we propose a new dataset for studying indirect answers. we provide both aggregated (ground truth/gold) annotations and the raw annotations. this allows us to study the effect of learning to integrate human disagreement into understanding indirect answers. our dataset, called friends-qia, was created by collecting question and answer pairs from transcripts of a popular tv series. the data collection differs in comparison to the recently introduced circa corpus. in their study, a set of 10 dialogue prompts were defined, and both questions and answers were collected by crowdsourcing @xcite . following this step, the annotation was crowdsourced again, resulting in a set of labels which was later conflated into a relaxed set of six classes. instead, we opted to collect the data from transcripts and manually annotate the question-answer pairs in house (by three of the authors of this paper, all highly proficient in english). examples from our dataset are provided in contributions in this paper, we a) introduce a new dataset, friends-qia, with 5,930 polar question-indirect answer pairs in english; foot_0 b) study the effectiveness of neural classifiers based on convolutional neural networks, both with traditional pre-trained word embeddings and contextualized bert embeddings; c) provide results on crossdataset evaluation, for which we train a model on both circa and friends-qia; and d) show that modeling human disagreement via deep learning from crowds is beneficial for this task. ## related work the task of understanding indirect answers in natural language processing is relatively new and has not been attempted by many yet. to enable progress on the task, @xcite created and released the first large-scale english language corpus, circa, consisting of 34,268 polar questions and their corresponding indirect answers. the difficulty of interpreting indirect answers is, however, widely studied in some early papers, (e.g. @xcite . following this work, de @xcite propose a logical inference model with probabilistic methods. they realize the influence of discourse conditions as well as the difference between the literal meaning of the answer and the interpretation by the two speakers. @xcite further underline the complexity of interpreting indirect answers to polar questions. they explore the existing edinburgh map task corpus @xcite which consists of two-person dialogues already coded for dialogue structure. @xcite recently also explore the understanding of indirect answers and report on the difficulty of the task. they create a new dataset, boolq, by combining search queries from the google search engine as questions and passages on wikipedia pages as answers. they attempt to classify such indirect answers by training bert-based neural models. @xcite further experiment with training bert models from scratch and by transfer learning from boolq, building on top of @xcite . their newly created dataset, circa, includes 10 question prompt types, spanning a wide variety of communication situations. their study shows promising results, and inspired our work. preparing a dataset for classification tasks often requires collecting labels from multiple annotators. a unanimous gold standard label cannot always be clearly achieved (e.g. @xcite @xcite . the problem of learning from multiple annotators has become more important and several attempts have been made to deal with biases present in such data. @xcite look into different levels of expertise among annotators and how a model can learn from this, taking into consideration the biases present while labeling the dataset. by measuring the inter-annotator agreement and incorporating the annotator uncertainty in model training, @xcite show that modelling annotator disagreement can be useful even in cases of seemingly more objective annotation tasks, like part-of-speech tagging. an interesting and similar approach was suggested by @xcite . their method, deep learning from crowds (dlfc), adds a crowd layer on top of a neural network which takes advantage of the reliability and bias from different annotators. it assumes access to the raw annotations from the training data. this is in contrast to the weighting approach proposed by @xcite , who integrate aggregated disagreement from a sample. as we have the full data available with multiple annotations in friends-qia, we here experiment with the deep learning from crowds (dlfc) approach proposed by @xcite . the key idea is to train a model directly from the noisy annotator-specific annotations modeled as additional per-annotator auxiliary tasks on top of the gold distribution. by using backpropagation and creating annotator-specific weights in the crowd layer, it allows us to include the label bias in the model's learning. there is emerging interest in learning from disagreement, and we refer the reader to a recent survey on this topic @xcite . ## friends-qia corpus we introduce a new corpus called friends-qia for studying indirect answers. the three steps involved in corpus construction are: data collection, data preprocessing and data annotation. instead of crowdsourcing the data collection as done by @xcite , we attempt to exploit already existing data by simply looking for useful question-answer (qa) pairs in dialogues. we use the existing transcriptions of the american televi- ## experiments we experiment with different variants of a convolutional neural network (cnn). in particular, we explore the use of bert embeddings, a crowd layer and combining datasets to obtain higher performance on friends-qia. base cnn our cnns are implemented with inspiration from @xcite , who uses 1dimensional convolutions in parallel. as shown in figure 2 , our implementation of the parallel convolutions consists of a convolutional 1d, a max pooling, a flatten and a dropout layer, after which they are concatenated and fed to the final output layer which uses softmax as activation function. the convolutional layer applies the relu activation function after which the maximum of each filter is selected. the dropout layer applies a dropout rate of 0.5 which is the same rate we use on the final layer when we apply regularization with a constraint of the l2-norm on the weights. the models either take english bert embeddings (devlin et al., 2019) as input or use glove embeddings @xcite . the models are optimized using the adam optimizer (kingma and ba, 2015). this general architecture is what all of our cnn variants are built upon. in preliminary experiments we took a portion of the training data as hyperparameter tuning set. we performed a grid search on the base cnn to find the optimal hyperparameters. for the cnn with bert no further hyperparameter tuning was done. crowd layer figure 3 illustrates the key idea of the dlfc approach we adopt on top of our cnn. training the models with a crowd layer is closely based on the paper and code by @xcite . following their implementation, our crowd layer is applied on top of the existing network. it takes as input the output of the dense layer and uses the annotator-specific labels, each modeled as a separate task, to propagate errors through the network and adjust the gradients. the layer is applied on the already trained and saved base model with an annotator-specific weight matrix (w r ), and further training is performed with the crowd layer. we found this setup to perform best. for the cnn+crowd layer model we use the same parameters as when training the base model. after training, the crowd layer is removed to allow for the final classification using the original dense layer with a softmax activation function. evaluation accuracy and macro-average f1score are the evaluation metrics. we first report results on the development set, and report the performance of the best models on the test portion. as baselines, we provide results of a majority baseline and a naive bayes model with word trigrams. ## results third, we observe that modeling the annotator uncertainty is beneficial. when we add the crowd layer to the base cnn (cnn + crowd layer), we observe both improved accuracy and f1-score, reaching 55.71 and 39.38, respectively. this is encouraging, as it shows that the disagreement in human annotations is informative for this task. finally, we observe that using bert embeddings as input representations consistently and remarkably improves the performance of the model. the cnn model with bert reaches an accuracy of 64.08 and f1-score of 49.16, which is an absolute improvement of over 9% in accuracy and over 16% in f1-score. what is, however, striking, is that this trend does not hold for the bert-based model trained on answer or question alone; we attribute this to the limited context, but this result warrants further investigation. especially the f1score is not even better than the base cnn model with glove embeddings on answers alone. nevertheless, bert representations overall improve the best models that take question-answer pairs as input, which are necessary for the overall best performance on the task. we observe that the crowd layer for the cnn taking the concatenation of question and answer as input (cnn) further improves upon bert representations, for which we obtain an f1-score of 55 and the overall best performance with the cnn with bert and crowd layer. the accuracy with the crowd layer does not improve considerably, which shows that the crowd layer is not equally helpful for all classes. we hypothesize that the crowd layer is particularly helpful for infrequent classes, which are the most difficult for this task and which we observed the highest annotator disagreement on. this is in fact the case, as we will discuss in section 6. training on friends-qia and circa we investigated whether we could improve the model further by using also the circa data as additional training data. the bottom rows of ## discussion in the following section, we provide additional insights on the task. ## conclusions and future work in this paper, we present friends-qia, a corpus for studying indirect answers in english dialogues. we propose to mine tv series transcripts of a wellknown tv series. recent work proposed this challenging answer-understanding task, and collected circa, a dataset on question-answer pairs from 10 dialogue prompts constructed instead using crowdsourcing @xcite . motivated by their work, we propose friends-qia. it contains a total of 5,930 question-answer pairs, and is released both with a majority label and the raw annotations. our results with cnns show that a model trained with bert embeddings outperforms a cnn trained with glove word representations. most interestingly, this is, however, only the case for a model that considers both question and answer. training on the answers alone provides reasonable signal for the task, but is not sufficient to resolve the indirect answer. understanding indirect answers is a challenging pragmatic task, and even human annotators might not agree on a single gold label. we experiment with a way to leverage disagreement in labeling, which proves encouraging: the macro f1-score of the two best cnn models further improves, when fine-tuning with a crowd layer that encodes individual annotator preferences. this is encouraging, as -to the best of our knowledge -human disagreement has not yet been leveraged in modeling for understanding indirect answers; our dataset can be considered a starting point for further research in learning from disagreement. overall, the data in friends-qia was originally written in a manuscript for the tv series, then uttered in a spoken dialogue context on the show and ultimately transcribed again. our results show that understanding indirect answers remains a challenging task. we are missing out on influ-ential factors such as intonation, body language and discourse relations from this dialogue context, when we only process the qa pairs in written form. modeling such factors is interesting and challenging future work.