Human-like informative conversations: Better acknowledgements using conditional mutual information (2021)

Abstract: This work aims to build a dialogue agent that can weave new factual content into conversations as naturally as humans. We draw insights from linguistic principles of conversational analysis and annotate human-human conversations from the Switchboard Dialog Act Corpus to examine humans strategies for acknowledgement, transition, detail selection and presentation. When current chatbots (explicitly provided with new factual content) introduce facts into a conversation, their generated responses do not acknowledge the prior turns. This is because models trained with two contexts - new factual content and conversational history - generate responses that are non-specific w.r.t. one of the contexts, typically the conversational history. We show that specificity w.r.t. conversational history is better captured by pointwise conditional mutual information (pcmi_h) than by the established use of pointwise mutual information (pmi). Our proposed method, Fused-PCMI, trades off pmi for pcmi_h and is preferred by humans for overall quality over the Max-PMI baseline 60% of the time. Human evaluators also judge responses with higher pcmi_h better at acknowledgement 74% of the time. The results demonstrate that systems mimicking human conversational traits (in this case acknowledgement) improve overall quality and more broadly illustrate the utility of linguistic principles in improving dialogue agents.

## introduction social chatbots are improving in appeal and are being deployed widely to converse with humans @xcite . advances in neural generation @xcite enable them to handle a wide variety of user turns and to provide fluent bot responses. people expect their interactions with these dialogue agents to be similar to real social relationships @xcite . in particular, they expect social conversational history (h) and new factual content (k), two largely independent contexts, are used to sample responses (g 1 , g 2 ) from a generative model. the samples differ qualitatively. while almost all of g 2 is verbatim from k (in gray), the first sentence in g 1 (in black) acknowledges using h and bridges to k. neural generation methods for adding new information @xcite @xcite measure progress using metrics like "engagingness", "appropriateness" and "informativeness". but these metrics are too broad and provide little actionable insight to drive improvements in these systems. on the other hand, psycholinguists and sociolinguists have studied human conversations in depth and have identified fine-grained conventions, principles and contracts @xcite @xcite . our first contribution is a linguistic analysis of how human conversations incorporate world knowledge. we manually annotate conversations from the switchboard corpus to identify key traits. in particular, we find that people apply four kinds of strategies: (1) acknowledgement of each other's utterances, (2) transition to new information, (3) appropriate level of detail selection and (4) presentation of factual content in forms such as opinions or experiences. to identify deficiencies of the above types in machine-learned models, we consider a simplified task of conversational rephrasing (figure 1 ), in which the factual content to be added is not left latent but is provided as a text input to the model ( as in dinan et al. (2019) ), along with conversational history. just as humans do not recite a fact verbatim in a conversation, we expect the model to rephrase the factual content by taking conversational context into account. we derive the data for this task using the topical chat dataset @xcite and fine-tune a large pre-trained language model on it. @xcite ; @xcite use maximum pointwise mutual information (max-pmi) to filter out bad and unspecific responses sampled from a generative language model. however, we observe that max-pmi responses lack in acknowledgement, an essential human trait. this is because a generated response that simply copies over the new factual content while largely ignoring the conversational history can have high mutual information (mi) with the overall input. our second contribution is a method to select responses that exhibit human-like acknowledgement. to quantify the amount of information drawn from the two contexts of new factual content and conversational history, we propose using pointwise conditional mutual information (pcmi). we show that responses with a higher pcmi w.r.t conversational history given factual content (pcmi h ) are judged by humans to be better at acknowledging prior turns 74% of the time. 1 then, we use pcmi h to identify max-pmi responses that lack acknowledgement and find alternative responses (fused-pcmi) that trade off pmi for pcmi h . despite a lower pmi, human annotators prefer the fused-pcmi alternative over the max-pmi response 60% of the time. 1 we release annotated conversations from the switchboard corpus (with guidelines), code for fine-tuning and calculating scores and human evaluations. foot_0 ## strategies for informative conversations to understand strategies used by humans while talking about factual knowledge, we annotate turns in human-human conversations. we adopt and extend herbert clark's approach to conversational analysis. according to his given-new contract @xcite , the speaker connects their utterances with the given information (assumed to be known to the listener) and adds new information. this builds up common ground @xcite between the two participants, defined to be the sum of their mutual, common or joint knowledge, beliefs and suppositions. we identify the following four aspects to the process of adding new information to a conversation. by step, connecting the given, stated information to new information. we annotate the semantic justifications for topical changes as different transition strategies. ## a method for richer acknowledgements current neural generation methods typically offer short and formulaic phrases as acknowledgements: "that's interesting", "i like that", "yeah, i agree". such phrases are appropriate almost everywhere and convey little positive evidence for understanding or grounding. the training corpus, on the other hand, contains richer acknowledgements, which generated responses should be able to emulate. we assume that the representational capacity of current neural models is sufficient and that out of all the sampled responses, some do indeed contain a richer form of acknowledgement. we posit that non-existent or poor sample selection strategies are to blame and that without a good sample selection strategy, improvements to the dataset, model or token-wise sampling methods are unlikely to help. we hypothesize that responses that are more specific to conversational history provide better evidence for understanding and hence contain richer acknowledgements. as a baseline sample selection strategy, we first consider maximum pointwise mutual information (max-pmi) (as used by @xcite ) between the generated response and the conversational contexts (i.e., new factual content and conversational history). however, this is insufficient because it is an imprecise measure of specificity w.r.t. conversational history. instead, we use pointwise conditional mutual information (pcmi) to maintain specificity with individual contexts and propose a combination of pmi and pcmi scores to select overall better quality responses than max-pmi. ## evaluation setup we derive the data for our conversational rephrasing task from the topical chat dataset @xcite . we use it to fine-tune a large pre-trained neural language model. this forms the base model as described in section 3. to evaluate our proposed methods, we design three experiments and perform a comparative study with human annotators. topical chat dataset this is a human-human chat dataset where crowd-workers were asked to chat with each other around certain topics. they were provided with relevant interesting facts from the "today i learned" (til) subreddit which they could use during the conversation. tils are are short (1-3 sentences), self-contained, interesting facts, most of them from wikipedia articles. when an utterance can be matched to a til (based on a tf-idf threshold of 0.12), we create an instance for the conversational rephrasing task: with the utterance as g, the two previous utterances as h and the corresponding til as k. we split the instances into training, validation and test sets (sizes in section a.1) such that all utterances related an entity belong to the same set. base model we use the gpt2-medium model (24-layer; 345m params) pretrained on the english webtext dataset @xcite , as implemented in huggingface's transfertransfo @xcite framework. fine-tuning is performed using the language modelling objective on the training set with default hyperparameters until lowest perplexity is reached on the validation set. during generation, we sample tokens using nucleus sampling @xcite with @xmath0.9 and temperature τ = 0.9 and get candidate responses. to compute auxiliary probabilities {p(g|h), p(g|k), p(g)} for these candidates, we use separate ablation models. the ablation models are trained similar to the base model but after removing respective contexts from the training inputs. ## results & analyses based on human annotations, we are able to reject h 0 in favor of h 1 in all three experiments ( figure 6 : attribution to acknowledgement span. a larger fraction of pcmi h can be attributed to human annotated acknowledgement spans compared to pmi(g; h) and pcmi k . to understand the mechanism behind the improvement in exp 3, we look at the distribution of samples w.r.t. pcmi k and pcmi h in figure 7 . we observe that max-pmi responses heavily skew the distribution towards higher pcmi k , whereas fused-pcmi responses show a more balanced improvement along both pcmi h and pcmi k . fused-pcmi increases both pcmi h and pcmi k (medians cross 75% quartiles), indicating that the responses are simultaneously specific to both h and k. ## discussion we show that samples with higher pcmi h provide better acknowledgement and fused-pcmi improves overall quality compared to max-pmi. thus, by improving acknowledgements -an aspect we identified during our analysis of human strategies -we were able to improve overall quality. this demonstrates the utility of linguistic analysis for finding interpretable and actionable metrics. while we show that our learnings apply to informative dialogue which adds factual knowledge @xcite pineau, 0 5 10 15 20 pcmi h 0 20 40 60 80 100 pcmi k all candidates max. pmi fused-pcmi figure 7: bivariate kernel density estimate plot all candidates, max-pmi responses and fused-pcmi responses. bivariate kernel density estimate plot w.r.t. pcmi k and pcmi h at levels 0.5 and 0.75. we see that fused-pcmi responses compared with max-pmi trade off little pcmi k for a large relative gain in pcmi h . see figure 8 in section a.2 for univariate box plots. there is scope for improvement: max-pmi still selects better responses than fused-pcmi in 40% of the instances. this could be because it is easy for the model to copy over k and generate a high pmi response that is also fluent and accurate. fused-pcmi encourages synthesis of acknowledgement using h and abstraction over k and it could therefore be prone to disfluencies and inaccuracies. we hope that orthogonal modeling improvements @xcite ) reduce such effects. a cause for concern with the human evaluation is low inter-annotator agreement for exp 1 and 3 where we ask them to pick responses with "overall better quality and suitability". however, quality measurements are inherently subjective; people differ in the importance they place on different aspects such as engagement, informativeness, fluency etc., as corroborated by prior work @xcite ) that shows low cohen's kappa (0.13, 0.22) for overall quality judgements. in this work, diverse expectations from multiple annotators are captured yet subsequently averaged into "overall quality". we leave it to future work to find finer-grained metrics that have high inter-annotator agreement and derive empirical weights to combine them into "overall quality". in this work, we looked at acknowledgements and their impact on quality in isolation, but in a real