N-best Response-based Analysis of Contradiction-awareness in Neural Response Generation Models (2022)

Abstract: Avoiding the generation of responses that contradict the preceding context is a significant challenge in dialogue response generation. One feasible method is post-processing, such as filtering out contradicting responses from a resulting n-best response list. In this scenario, the quality of the n-best list considerably affects the occurrence of contradictions because the final response is chosen from this n-best list. This study quantitatively analyzes the contextual contradiction-awareness of neural response generation models using the consistency of the n-best lists. Particularly, we used polar questions as stimulus inputs for concise and quantitative analyses. Our tests illustrate the contradiction-awareness of recent neural response generation models and methodologies, followed by a discussion of their properties and limitations.

## introduction recent advanced response generation models @xcite @xcite can generate relevant and meaningful responses, which can resolve dull response problems @xcite @xcite . this advancement reveals additional flaws in the quality of neural model responses, such as contradiction. contradiction is a critical error in dialogue because a single contradictory response can disrupt the flow of the dialogue @xcite . a generation model outputs a response by selecting the candidate with the highest likelihood (1best) from an n-best candidate list. prior work has demonstrated that generating the n-best lists with noncontradictory 1-bests is an open challenge @xcite @xcite . thus, one practical technique for avoiding contradiction is to have an accurate contradiction detector that eliminates all contradictory candidates from the nbest list @xcite . in this scenario, the consistency of all candidates in the n-best list, not just the 1-best, substantially impacts whether the final output is contradictory because the final response is chosen from the n-best list. nonetheless, earlier quantitative investigations of contradiction relied solely on 1-bests from models @xcite . in this study, we analyze the n-best lists generated by the models to explore methods for enhancing neural response generation to avoid contradiction. specifically, we first consider how analyzing an n-best list should be approached. then, we propose a method for statistically analyzing the n-best lists (figure 1 ). since it is impractical to study all conceivable contradictions in a dialogue, we first focus on contradictions in response to polar questions. 1 we use our method to highlight the contradiction-awareness of recent highperformance neural response generation models and methodologies. our results show that beam search has limitations in terms of avoiding contradiction and that the newer techniques, such as unlikelihood training @xcite , can help overcome these limitations. ## analysis perspectives first, n-best lists must be generated to prevent contradiction, assuming the filters can remove contradictory responses. an ideal model produces output that is noncontradictory and outperforms in many other criteria, such as relevance or informativeness. a model must generate at least one noncontradictory candidate to deliver a noncontradictory output. furthermore, even noncontradictory candidates could be eliminated based on other criteria (e.g., relevance, informativeness). therefore, it can be hypothesized that having more noncontradictory responses in an n-best list would enhance the final output quality across various criteria. taking the above into account, we examine n-best lists based on the certainty of the existence of noncontradictory responses (certainty), and the variety of noncontradictory responses (variety): given a set of inputs q, we calculate them as follows: ## analytical inputs and evaluation to analyze a model from the aforementioned viewpoints, we consider how to prepare the analytical inputs and evaluate the generated responses in this section. ## experiments we demonstrate how our framework shows the properties of n-best lists, which could be quite influential in terms of avoiding contradiction. we demonstrate this by comparing the n-bests generated by conventional beam search (bs) versus recently proposed techniques. ## conclusion based on the recent development of contradiction detectors, removing contradictory candidates from models' n-best lists is a practical method for avoiding contradiction. in this method, the consistency of all candidates in the n-best lists substantially affects whether the final outputs are contradictory. we quantitatively examined the properties of the n-best lists in terms of avoiding contradiction, using polar-typed questions as analytical inputs. we demonstrated that the proposed framework exhibits the properties of n-best lists based on certainty and variety. certainty determines whether an nbest list has at least one noncontradictory response, whereas variety evaluates how many noncontradictory responses each n-best list has. the results, particularly, demonstrated the present limitations on achieving high certainty and variety when using the well-established beam search method. in addition, our method emphasizes the improvements in certainty and variety achieved by recently proposed response generation strategies. our approach, which analyzes models' n-best lists based on certainty and variety, can be applied to any response generation problem, not just polartyped response generation, which will be future work.