On the True Distribution Approximation of Minimum B ayes-Risk Decoding (2024)

Abstract: Minimum Bayes-risk (MBR) decoding has recently gained renewed attention in text generation.MBR decoding considers texts sampled from a model as pseudo-references and selects the text with the highest similarity to the others.Therefore, sampling is one of the key elements of MBR decoding, and previous studies reported that the performance varies by sampling methods.From a theoretical standpoint, this performance variation is likely tied to how closely the samples approximate the true distribution of references.However, this approximation has not been the subject of in-depth study.In this study, we propose using anomaly detection to measure the degree of approximation.We first closely examine the performance variation and then show that previous hypotheses about samples do not correlate well with the variation, but our introduced anomaly scores do.The results are the first to empirically support the link between the performance and the core assumption of MBR decoding.

## introduction minimum bayes-risk (mbr) decoding has recently re-emerged as a better alternative to beam search in text generation such as neural machine translation (nmt), text summarization, and image captioning @xcite @xcite @xcite . mbr decoding first samples texts from a model and then selects the text most similar to the others, considering the text samples as substitutes for references. therefore, sampling plays an important role in mbr decoding, and previous studies have reported that the performance varies by sampling methods @xcite @xcite . from a theoretical standpoint, the samples are assumed to approximate the true distribution, the distribution of human-quality translations @xcite . if the approximation deviates, biases can emerge in results of mbr decoding. this implies a significant link between the performance variation and approximation quality. although previous studies explained the performance variation by some properties of samples, e.g., sampling bias and cumulative probability mass @xcite , those properties have no clear relation with the true distribution. consequently, the relation between the performance gains by sampling methods and the core assumption remains unclear. this study aims to empirically support the link between the performance and the approximation of true distribution. to this end, we introduce measures for the degree of approximation. if the assumption for samples holds, references, which are drawn from the true distribution by definition, should not deviate from the majority of the samples. based on this recasting, we employ anomaly detection (also called outlier or novelty detection) for the measure. our hypothesis is that references achieve lower anomaly scores among samples obtained with a higher-performance sampling method. we first closely examine the performance variation by sampling methods. then, we show that the variation highly correlates with the anomaly scores but not so with the properties based on previous hypotheses. the results are the first to provide empirical evidence for the link between the performance and core assumption, which is an important step to understanding the connection between the actual performance and the theory of mbr decoding. ## preliminaries let u(y, r) be a utility function to measure the quality of model translation y (candidate; freitag et al., 𝑦 " : the bird is flying. ## performance variation by sampling previous studies reported that performance varies by sampling methods in nmt. however, they used the same set of model translations for both candidates and pseudo-references @xcite @xcite or explored sampling methods only for candidates @xcite . these settings obscure the effect of pseudo-references, for which the true distribution is assumed, on the performance variation. this section shows the effect of pseudoreference sampling on performance by evaluating pseudo-references separately from candidates. ## hypotheses for performance variation the previous section confirmed that the performance varies by sampling pseudo-references. the question that naturally arises in response to the results is: why does this variation occur? different splits to explore and test the combination to ensure the generalization. nevertheless, our subsequent results in tables 1 , 2 , 4, and 5 suggest the generalization of the found best combination as it consistently performs the best across almost all language pairs. epsilon (ϵ = 0.02) ## experiments we test the hypotheses discussed in the previous section by evaluating the correlation between the performance variation and the properties or anomaly scores. ## conclusion this study investigated the relation between the performance of mbr decoding and the core assumption about samples: samples follow the true distribution of references. we introduced anomaly scores used in anomaly detection to evaluate the approximation of the true distribution. experimental results demonstrated that the anomaly scores correlate with the performance significantly better than the properties hypothesized to explain the performance variation in prior literature. the previous hypotheses assumed that unbiased sampling (avg. prob.), diverse and probable samples (cum. prob.), or high expected utility (cand. and ref. sim.) are the key properties of samples to achieve high performance. however, these properties do not have an obvious relationship to approximating the true distribution of references, in contrast to the anomaly scores we employed. these results show the insufficiency of existing hypotheses about the properties that samples should possess. the results are also the first to empirically support the link between the actual performance and the key assumption of mbr decoding. we believe this serves as an essential step to understanding the connection between the actual performance and the theory of mbr decoding.