Measuring and Improving Semantic Diversity of Dialogue Generation (2022)

Abstract: Response diversity has become an important criterion for evaluating the quality of open-domain dialogue generation models. However, current evaluation metrics for response diversity often fail to capture the semantic diversity of generated responses, as they mainly consider lexical aspects of the generated responses. In this paper, we introduce a new automatic evaluation metric to measure the semantic diversity of generated responses. Through human evaluation, we demonstrate that our proposed metric captures human judgments on response diversity better than existing lexical-level diversity metrics. Furthermore, motivated by analyzing an existing dialogue dataset, we propose a simple yet effective learning method that improves the semantic diversity of generated responses. Our learning method weights training samples based on the semantic distribution of the training set.We show that our learning method improves response diversity and coherency better than other baseline methods through automatic and human evaluation.

## introduction open-domain dialogue generation @xcite has greatly progressed with the development of large-scale pretrained language models @xcite in the last decade. however, although dialogue generation models can produce fluent responses, they are also known for frequently generating dull and uninformative generic responses (e.g., "i don't know"), degrading their engagingness @xcite . to alleviate this problem, many studies @xcite @xcite have been conducted to enhance the diversity of generated responses, and response diversity has become an important criterion for evaluating the quality of generated responses. the current evaluation protocol has relied on lexical-level evaluation metrics such as distinctn (dist-n) @xcite and entropy-n (entn) @xcite to measure the diversity of generated responses. however, according to recent studies @xcite , these lexical-level evaluation metrics often fail to capture semantic diversity since responses including similar words can have different semantics and responses with different words can have similar semantics @xcite .