Rethinking and Refining the Distinct Metric (2022)

Abstract: Distinct is a widely used automatic metric for evaluating diversity in language generation tasks. However, we observed that the original approach to calculating distinct scores has evident biases that tend to assign higher penalties to longer sequences. We refine the calculation of distinct scores by scaling the number of distinct tokens based on their expectations. We provide both empirical and theoretical evidence to show that our method effectively removes the biases existing in the original distinct score. Our experiments show that our proposed metric, Expectation-Adjusted Distinct (EAD) , correlates better with human judgment in evaluating response diversity.To assist future research, we provide an example implementation at https://github.com/lsy641/Expectation-Adjusted-Distinct .

## introduction the diversity of generated texts is an important evaluation aspect for dialogue generation models since most dialogue models tend to produce general and trivial responses (e.g. "i don't know" or "me too") @xcite . several metrics have been proposed to evaluate the text diversity, and the distinct score @xcite is the most widely applied metric due to its intuitive nature and convenient calculation. it has become a de facto standard to report the distinct score to compare the performance of different models in terms of response diversity @xcite @xcite @xcite @xcite @xcite . most previous works follow the initial approach of @xcite to calculate the distinct score, i.e., dividing the number of unique tokens in the figure, "natural" means that text sets are sampled from a real corpus while "designated" means that the sets are sampled from a designated distribution. see details in section 2. we argue that the scaling factor of distinct requires a comprehensive discussion for two reasons. first, prior research in non-computational linguistics has demonstrated the shortcomings of distinct's scaling approach @xcite . we found that early applications of distinct exist in psycholinguistics, where researchers leveraged this metric to assess the language diversity of children with communication disorders @xcite . their research showed that as a child speaks more words, distinct experiences an adverse decline since each extra word that the child utters adds to the total number of words, yet it would only increase the number of distinct words if the word had not been used before @xcite . second, we also discovered an uncommon decline of this metric on both a natural corpus and a designated distribution sampler when the total num-ber of words increases. as illustrated in figure 1 , the original distinct cannot produce a stable value and experiences a sharp decrease with increasing utterance length in both natural and designated distributions. however, as a qualified metric needs to support quantitative comparison among different methods, its value should stay invariant when the distribution of the words appearing is determined. this result is consistent with the findings of psychologists, indicating an unfair penalty does exist in such a scaling method. our contributions are summarized as follows: ## preliminary discussion about original distinct to demonstrate the shortcoming of the original distinct, we illustrated the normalised distinct scores on two types of texts at different lengths (figure 1 ). the first type of text is sampled from an artificially designated distribution while the other is sampled from a natural language corpus. in detail, we adopted in our experiments, we use bert's vocabulary's size (@xmath0. in addition, we leveraged opensubtitles 1 as our natural language corpus. for each length, we sampled 2000 sentences as a set and calculated scores of each set. as shown in figure 1 , we observe that the original distinct scores decrease sharply with increasing utterance length in both distributions. we can observe that given the same distribution of words (original-designated), lengthier texts will get lower scores than shorter texts. we highlighted this problem because it is extremely simple for models to control the length of texts by using decoding tricks, e.g. adjusting the penalty coefficient (vijayakumar 1 http://opus.nlpl.eu/opensubtitles2018.php @xcite . in such cases, it might seem that a model has outperformed other models on this metric. however, as shown by our experiments, it is not reasonable to assume that this model generates more diverse responses. the same observation can be made for the natural language corpus (original-designated). as language distribution is more complex than what we are able to formulate, we depicted the performance of the original distinct on 6 famous datasets in appendix. these cases indicate that the original distinct is not a suitable metric for evaluating diversity. ## ead in practice as ead is based on the idealized assumption that does not take language distribution into account, we further discuss this problem and propose a potential practical way of expectation-adjusted distinct in real situations. before applying ead, it is necessary to explore the relationship between score and text length (figure 1 ) and check the performance of ead on the training data. to our knowledge, if the training data is from large-scale open-domain sources such as opensubtitles and reddit, ead can maintain its value on different lengths. hence, it can be directly used for evaluating models trained on these datasets. however, we found our experiments on datasets such as twitter showed a decline in ead on lengthier texts. this is probably because input length limitations on these platforms (e.g. 280 words on twitter), which induces users to say as much information as possible within a shorter length. in these situations, it is unfair to use ead to evaluate methods that tend to generate lengthier texts. there exist other metrics for evaluating diversity but none are as widely-used as distinct @xcite . specifically, self-bleu proposed by @xcite is extremely timeconsuming as its computation complexity is o(n 2 ), where n denoted the size of the test set. ## conclusion in this paper, we present an improved variation of the distinct metric, which is a widely-used measure for evaluating response diversity in dialog systems. we provide the theoretical formulation and empirical evaluation of our proposed metric (expectation-adjusted distinct). the results demonstrated that expectation-adjusted distinct has a higher correlation with human evaluation in comparison with other metrics. the proposed metric is not limited to dialogue generation models but also suitable to evaluate text generation tasks where diversity matters. where theoretically, expectation-adjusted distinct can have values larger than 1 (e.g. when @xmath1, which is an extremely rare case in practice: as we utilized the upper bound for measuring the expectation, it is exceptionally hard for n to obtain an equal value to or an even greater value than e( nupper ).