A uto R eply: Detecting Nonsense in Dialogue with Discriminative Replies (2023)

Abstract: We show that dialogue models can detect errors in their own messages, by calculating the likelihood of replies that are indicative of poor messages. For example, if an agent believes its partner is likely to respond “I don’t understand” to a candidate message, that message may not make sense, so an alternative message should be chosen. We evaluate our approach on a dataset from the game Diplomacy, which contains long dialogues richly grounded in the game state, on which existing models make many errors. We first show that hand-crafted replies can be effective for the task of detecting nonsense in applications as complex as Diplomacy. We then design AutoReply, an algorithm to search for such discriminative replies automatically, given a small number of annotated dialogue examples. We find that AutoReply-generated replies outperform handcrafted replies and perform on par with supervised learning approaches.

## introduction detecting nonsensical dialogue generation has been an enduring challenge in dialogue research @xcite @xcite . previous work proposed datasets with bad message annotations and built classification models to detect them. but such a supervised learning approach requires building an extra model without fully utilizing the dialogue model's own language ability. in complex dialogue applications, data annotation is often limited but the space of bad messages is large, ranging from easy (e.g., repetition), grounding-related (e.g., contradiction to context) to challenging even for human novices (e.g., domainspecific mistakes). in this paper, we refer to all the mistakes made by the dialogue model as nonsense. nonsensical message and follow-up reply france: i won't move out of spain, just don't move out of marseilles italy: thank you. you will be watching my moves. france: ok italy: :) france: are you moving to spain, spain? italy: i don't understand your message. previously, @xcite combined the probabilities of hand-crafted replies like "that's different from what you said", "wow! very interesting." to get a score for dialogue quality evaluation. inspired by this, we utilize the dialogue model itself and develop a reply-based approach that uses the probability of discriminative follow-up replies to detect if a message is nonsensical. intuitively, if a message doesn't make sense like in however, like prompt engineering, manually designing such replies is labor intensive and hard to scale to broad classes of errors. therefore, we propose autoreply, which uses the dialogue model itself and a small set of nonsense-annotated dialogue examples, and does not require hand-crafting replies or training an external supervised classifier. autoreply is a contrastively-pruned search procedure that finds discriminative replies by iteratively selecting tokens that are highly likely after multiple nonsensical messages. the motivation is that, if a given token is highly likely after multi-ple nonsensical messages, then it likely captures nonsense-related features and may lead to discriminative replies. we enumerate and expand such tokens to generate replies, and then contrast the generated replies against good messages to prune the search and keep the discriminative replies only. we evaluate our reply-based approach on dialogue models for the board game diplomacy @xcite . diplomacy involves lengthy dialogues in which players attempt to persuade others to take certain actions in the game @xcite . this setting represents a challenge to existing dialogue models, as messages often contain detailed discussions of both past events and hypothetical future actions in the game, providing many opportunities for subtle errors in messages. even large pre-trained models, and non-expert humans, make frequent mistakes in messages-and discriminating good from bad messages often requires complex reasoning. to summarize, our contributions are three-fold. first, we propose to utilize the dialogue model itself and discriminative replies to detect the models' own mistakes and demonstrate that such an introspective reply-based approach is effective in diplomacy, a complex dialogue setting. second, to reduce the manual work in reply design, we propose autoreply to automatically generate a large number of discriminative replies from limited annotations. third, experiments show that autoreply can automatically generate many high-quality discriminative replies, and these replies achieve substantially better results than hand-crafted replies, and perform on par with large fine-tuned supervised models without actually training extra classifiers. we note that our main goal is not to build stateof-the-art nonsense detector but rather, to explore a drastically new approach to dialogue nonsense detection, which utilizes the dialogue model's own ability to introspect, roll out to the future, and detect its own mistakes without external classifiers or new parameters. we show promising results and hope to inspire more research in similar directions. ## methods we propose to detect whether a message x from a dialogue language model l is nonsensical by using the model's own distribution over possible reply messages r: p l (r|x). 1 we first describe how to detect nonsense with a set of hand-crafted replies in section 2.1, and then describe how to automatically generate replies with autoreply in section 2.2. ## diplomacy and data collection we evaluate our method on diplomacy, a sevenplayer board game where each player controls the units (fleets and armies) of an european power starting in the year 1901, with the goal to win as many supply centres as possible. in appendix b, we briefly describe the game rules. one game consists of many years, and each year is divided into phases between which, the players are permitted to communicate with each other, and thus the conversations are very long: on average, each training example's dialogue history contains 140+ messages richly grounded on the game state (see in summary, diplomacynonsense is a challenging dataset, in which detecting nonsensical message can involve both reasoning over long dialogue contexts and grounding in a rich environment. see appendix a.3 for more details on the dataset. ## experiments in our experiments, we fine-tune a case-insensitive bart (lewis et al., 2019) l on human-human diplomacy dialogues from webdiplomacy and use it to both score and generate the follow-up replies. for more experimental details, please refer to section a.3. we compare autoreply's classification results against various baselines for nonsense detection and analyze the generated replies qualitatively. we also compare it against a supervised model that requires training a large-scale classifier. now we describe the models. hand-crafted, is a baseline classifier built with the hand-crafted replies, as described in section 2.1. l-generated replies, where we use the language model l directly to generate 20 replies for each bad example and ensemble them for a simple baseline. l-generated replies are often specific to one single bad example and ignore the nonsenserelated aspects, e.g., "can you please support albania to greece?", "russia lied to me, as he is working with turkey. we can win if we work together.", etc. for the threshold, for a fair comparison with autoreply, we also use the maximum log probability over good training examples ## related work detecting nonsensical dialogue is a well-known challenge @xcite . previous research formulated nonsense detection as a supervised learning problem @xcite . but as collecting such datasets can be costly and difficult to scale, recent work proposes to evaluate generated text with prompt-based learning. @xcite proposed bartscore and showed that the probability of hand-crafted prompts can be used for text quality evaluation. mehri and eskenazi (2020) used dialogpt @xcite to get the probability of hand-crafted follow-up replies to evaluate the generated dialogues. we also use follow-up replies to detect bad messages, but instead of hand-crafted replies, autoreply automatically generates many followup replies to cover different nonsense types. our work is one type of "prompt-based learning", which utilizes pretrained language models and text prompts for downstream nlp tasks @xcite . while prompts can be manually designed @xcite , recent work has proposed prompt generation @xcite to automatically obtain high-quality prompts at scale. for instance, @xcite proposed a gradient-guided method to search for tokens that would trigger certain targets, which requires many training examples. @xcite used limited annotated examples to predict tokens at specified positions in the template. different from previous work, our goal is to generate free-form follow-up replies that are discriminative: likely after nonsensical messages but unlikely after good messages. diplomacy has been a long-standing ai benchmark game due to its large action space and com-plex communication between players. many relevant previous studies focused on the "no-press" variant of the game, where communication is not permitted @xcite . @xcite al. (2019) research on full-press diplomacy, the variant that permits communication, is gaining more popularity. @xcite studied the use of lies in diplomacy and collected a dataset with truth and lie message annotations. @xcite built the first human-level ai agent and ranked in the top 10% players in an online league. in this paper, we focus on diplomacy as a complex testbed for nonsensical dialogue message detection. ## conclusions and discussion in this paper, we propose to detect nonsensical messages using the probability of the dialogue model's own follow-up replies like "i don't understand", without building an extra classifier. we evaluate this reply-based approach on diplomacy, a complex board game with rich verbal communication dynamics. we first show that hand-crafted replies are effective for nonsense detection. to reduce the labor of reply engineering, we develop autore-ply, a search algorithm to automatically generate discriminative replies. experiments show that autoreply can generate many high-quality discriminative replies and achieves significantly better performance than the hand-crafted baselines, and performs on par with large supervised models. one thing to note is that our reply-based approach and autoreply are not limited to diplomacy, and can be applied to various other dialogue problems, such as improving dialogue safety by detecting offensive language (with replies like "that's not nice to say") or detecting factual contradictions. for future work, please refer to section a.1.