Mark-Evaluate: Assessing Language Generation using Population Estimation Methods (2020)

Abstract: We propose a family of metrics to assess language generation derived from population estimation methods widely used in ecology. More specifically, we use mark-recapture and maximum-likelihood methods that have been applied over the past several decades to estimate the size of closed populations in the wild. We propose three novel metrics: \textrm{ME}_{\textrm{Petersen}} and \textrm{ME}_{\textrm{CAPTURE}} , which retrieve a single-valued assessment, and \textrm{ME}_{\textrm{Schnabel}} which returns a double-valued metric to assess the evaluation set in terms of quality and diversity, separately. In synthetic experiments, our family of methods is sensitive to drops in quality and diversity. Moreover, our methods show a higher correlation to human evaluation than existing metrics on several challenging tasks, namely unconditional language generation, machine translation, and text summarization.

## introduction population estimation methods have been widely used in ecology to study the development of species over the last several decades @xcite ). existing population estimation methods focus on open populations, where births, deaths, and migrations are taken into account, or closed population methods, where the population is assumed to remain static over the population estimation study. in this work, we focus on closed population methods and study how their respective population estimates can be used to evaluate an evaluation set of generated samples, given a reference set of real samples. in this work, samples are either contextualized word or sentence embeddings. we study two markrecapture methods, namely the petersen @xcite and the schnabel @xcite estimators, where samples are captured and marked, released, and recaptured. the number of such different samples is then used to estimate the population size. we further use capture @xcite , a maximumlikelihood method, which uses the number of marked samples over multiple captures for such estimation. accurate evaluation of generated data is essential to correctly measure in what degrees we can improve the overall generation process. depending on the use case, single-valued metrics may suffice to assess specific conditional language generation tasks, such as machine translation and text summarization, where we are interested in evaluating the similarities of a generated translation or summary to a specific reference translation or summary, respectively. on the other hand, on unconditional language generation, for example, it may be useful to have separate measures for the diversity and quality of the generated set, enabling the identification of possible shortcomings of our generation system and try to fix it accordingly. this has been an active area of generative models research @xcite , with several works focusing on stimulating diversity while maintaining the overall sample quality @xcite @xcite @xcite . mark-evaluate (me) is a family of 3 novel language evaluation methods based on the above population estimation methods: me petersen and me capture retrieve a single-valued metric to assess an evaluation set, while me schnabel returns a double-valued metric, separately measuring the quality and diversity of the evaluation set. our main contributions can be listed as follows: (i) proposal of 3 novel language metrics (section 3) that are sensitive to mode collapse (section 4.1) and quality detriment (section 4.2) and show a high correlation to human evaluation on challenging text generation tasks, such as unconditional language generation (section 5), machine translation (section 6.1) and text summarization (section 6.2). (ii) in-depth study of the language assessment capability of popular existing metrics, i.e. fid @xcite , @xcite and impar @xcite , primarily used to evaluate image generation in the past. (iii) usage of contextual information and different levels of granularity to assess language, by using either contextualized word (sections 6.1 and 6.2) or sentence embeddings (sections 4, 5) derived from bert @xcite . (iv) code for the reproducibility of the results will be publicly available. ## related work while acknowledging the importance of traditional evaluation metrics, such as bleu @xcite , rouge @xcite and meteor @xcite , we will focus on the new trend of unsupervised methods that use embedding representations of pre-trained models to assess a set of evaluation samples. our family of methods analyzes the data manifold to assess the evaluation set by using k-nearest neighbors to determine the capture volume. several methods have been recently proposed to assess data generation using topological information, however, they were primarily intended to assess image generation @xcite @xcite . in this work, we investigate the performance of such methods in the text domain, analyzing their behavior on synthetic experiments and their correlation with human evaluation. precision and recall for distributions (prd) was proposed by @xcite and uses of kmeans @xcite to build histograms of the discrete reference and evaluation distributions over the clusters' centers. the evaluation distribution is then assessed in terms of relative probability densities. precision is obtained by calculating the probability of an evaluation sample falling within the reference distribution's support. on the other hand, recall is retrieved by calculating the probability of a reference sample falling within the evaluation distribution's support. @xcite suggested several improvements to the above method, which we call improved precision and recall (impar). first, instead of k-means, they proposed to use k-nearest neighbors to approximate the reference and evaluation manifolds by building a hypersphere around each sample to its k-th nearest neighbor. second, they simplify prd's notions of precision and recall, by calculating the probability of an evaluation sample to fall within at least one reference sample's hypersphere, and vice-versa, respectively. the proposed manifold approximations by the usage of hyperspheres present a simple, yet effective way of representing the reference and evaluation manifold in an explicit, nonparametric way. we build upon this idea and use identical hyperspheres to determine the capture volume used by our different estimators to estimate the population size. fréchet inception distance or fid @xcite is a widely used single-valued metric that assesses data similarity by calculating the distance between the reference and evaluation distributions. even though originally proposed for the image domain, @xcite adapted fid to evaluate text generation by getting vector representations from infersent @xcite , instead of inception-v3 @xcite . even though this metric precedes prd and impar, fid is still commonly used to assess generative models in the image and text domain. as previously mentioned, we study both the usage of sentence embeddings, derived from sbert @xcite , as well as contextualized word embeddings from bert @xcite , which have been recently shown to improve language assessment, both in a supervised @xcite and unsupervised manner @xcite . more specifically, bertscore measures precision and recall from a reference and evaluation text by calculating the required transport of each word of a given text to the most semantically similar word in the other text. on the other hand, moverscore measures the semantic distance between two texts by calculating the minimum transport required between the reference and evaluation texts. these metrics are ideal for conditional text generation, such as machine translation and text summarization, where an evaluation text should match a given reference text. figure 1: each sample @xmath0, respectively. marked samples are represented by filled circles. in this illustration, hyperspheres reach to each sample's nearest neighbor of the same set, i.e. @xmath1. for me petersen and me schnabel , we first capture and mark all samples inside any hypersphere of s. then, for me petersen , we count the number of marked samples or recaptures inside any hypersphere of s . for me schnabel , we perform a similar process iteratively, marking and recapturing samples inside the hypersphere of each s , resulting in all samples being marked in the end. on the other hand, me capture captures and marks samples inside each hypersphere of s and s . ## mark-evaluate in this work, we consider population estimation methods for closed populations, where the true population size remains constant throughout the estimation study. in our use case, our population consists of two sets, namely a reference set s r and an evaluation set s e . the true population size (p ) is then known a priori and represents the total number of samples in the two sets: given an estimated population size ( p ) from one of the used estimators, we measure the accuracy loss (a) as follows: with a low accuracy loss, i.e. a(p, p ) ≈ 0, representing a good population estimate, and a high accuracy loss, i.e. a(p, p ) ≈ 1, otherwise. our population estimation methods assume all samples to have an equal chance of capture, which is influenced by our capture volumes: hyperspheres that reaches each reference or evaluation sample's k-th nearest reference or evaluation neighbor, respectively. hence, if evaluation samples tend not to be inside any reference sample's hypersphere and vice-versa, the population estimate will likely be poor due to the lack of captured samples in the estimation study. our methods can be separated into three categories: single marking and recapture (me petersen ), multiple markings and recaptures (me schnabel ) and multiple markings and captures (me capture ). let us consider two sets of samples s and s , where each sample is a contextualized word embedding or a sentence embedding derived from bert, depending on the task. figure 1 illustrates our family of methods. adapting @xcite 's formulations, we define a binary function f that returns whether a sample @xmath2: the petersen estimator @xcite , relies on a single marking step and a single recapture step. it merely assumes that the ratio of marked samples (m ) in the marking step and the population size (p ) is equivalent to the ratio of recaptured samples (r) and captured samples (c) in the recapture step. the population size estimate ( p petersen ) is then calculated as follows: the petersen estimator was extended by @xcite to incorporate multiple markings and recaptures. the population size estimate ( p schnabel ) is calculated from t consecutive petersen estimates: the set of marked samples at each iteration @xmath3, . . . , t }, can be defined recursively as: me schnabel 's first marking step is identical to me petersen 's single marking step (m (1, s, s ) = m (s, s )), with all samples in s as well as samples in s that are inside at least one hypersphere of s being marked. by the final marking step, all samples will be marked since we iterated through all of them: m t (s, s ) = |m (t, s, s )|. for the other iterations, 1 < @xmath4, samples in s that are captured, i.e. are k-nearest neighbors of the s being iterated, but are not yet marked, are added to the marked set. after all recapture steps, which excludes the first marking step, the number of captured samples will be the number of samples in s and their respective k'th nearest neighbors as well as samples in s that are inside the hypersphere of each s : both me petersen and me schnabel are mark-recapture methods since they rely on marking and recapturing information to estimate the population size. we further used a maximum log-likelihood method: the model null from program capture @xcite . by considering the total number of marked 1 10 20 30 40 number of neighbors (k) 20000 21000 22000 23000 population estimate (p) mnli population size (p) p petersen p schnabel p capture 1 10 20 30 40 number of neighbors (k) 0.85 0.90 0.95 1.00 score mnli me petersen me schnabel me capture samples (m t ) and the total number of captures (c total ) over t iterations, with @xmath5, we iterate through several provisional population estimates (p @xmath6: the total number of captures corresponds to the number of samples in s and s and their respective neighbors, as well as the number of samples in s inside the hypersphere of a given s and vice-versa: the final population estimate ( p capture ) is then the estimate that maximizes equation 6: our family of methods uses the accuracy loss of each estimator to compute their scores as follows: note that, due to its iterative nature, me schnable may be used to separately assess the quality and diversity of an evaluation set s e given a reference set s r . more specifically, quality may be calculated by me schnable (s r , s e ), whereas diversity may be measured by me schnable (s e , s r ). on the other hand, me petersen and me capture are single-valued metrics, since me petersen (s r , s e ) = me petersen (s e , s r ) and me capture (s r , s e ) = me capture (s e , s r ). we refer to the appendix for theoretical discussions. to study the effects of different capture volumes, determined by different k, we used sbert to get the sentence embeddings of 10k training sentences from mnli @xcite as the reference set, and 10k validation sentences as the evaluation set. results are shown in figure 2 , with @xmath7, . . . , 40}. we observe that as k increases, the population size estimated by all estimators converges to the true population size. in turn, the scores of our family of methods also converge to their maximum value of 1. ## synthetic experiments the sentence-pair information for these experiments, we treated each sentence independently. our reference set consists of sentences from the training set, whereas our evaluation set has sentences from the validation set. we kept the size of the reference and evaluation sets equal throughout our experiments to reduce possible method instabilities regarding sample size. we follow the experiments in @xcite and simulate diversity loss by dropping sentences from certain topics (section 4.1), whereas quality detriment is induced by swapping the words of each sentence (section 4.2). for both experiments, we use sbert embeddings from bert-base pre-trained on snli @xcite and mnli @xcite datasets ( 'bert-base-nli-mean-tokens' foot_0 ). ## language generation we further assessed the text generated by ten different language generation models presented in @xcite . the models include a traditional language model and several types of autoencoders, namely variational, adversarial, adversarially regularized, and plain autoencoders. we used the human ratings assigned to each model's fluency presented in their work to study the correlation of our family of methods and other tested metrics to human evaluation. reverse and forward cross-entropy, i.e. reverse ce and forward ce, have been commonly used to assess text generation in the past @xcite @xcite . the reported reverse ce and forward ce results were taken from @xcite , obtained by training a language model on english gigaword @xcite . we refer to @xcite for additional details. we used sbert embeddings from bert-large trained on snli and mnli datasets ('bert-large-nlimean-tokens') since they achieved the best-reported performance in @xcite . note that, since human evaluation is only related to each model's fluency, we only report the quality assessment scores for me schnabel , prd, and impar. for our family of methods, as well as prd and impar, we iterate through k values until correlation drops and present the results with the best k of each method. overall, our family of methods achieves the highest correlations to human evaluation. note that despite being outperformed by fid, me petersen still outperforms prd and impar across all correlations. additional results with default k for our family of methods, prd, and impar, as well as a comparison with infersent and different sbert embeddings, are provided in the appendix. ## contextualized word embeddings we will now shift our focus to conditional language generation under finer-grained representations, i.e. contextualized word embeddings. we used embeddings from bert-base fine-tuned on mnli, identically to @xcite . for a fair comparison, we used the same embedding representations for all the methods in the following experiments. due to the likely imbalance of reference and evaluation samples, we only report the quality assessment or precision of double-valued metrics. similarly to section 5, we report the results with the best k. additional results with default k can be found in the appendix. using the information of the last layers of bert has been shown to help in several downstream tasks @xcite . this has also been shown for language assessment, observed by, for example, the fact that the best performing layers of bertscore are often latter layers @xcite . moverscore extends this thinking and aggregates the representations of the last five layers of bert with p-means. for our methods, instead of aggregating or routing this information, we use the vector representation from the last five layers for each specific word. thus, each word has five representations, defined as five samples, in our scheme. see figure 5 for an illustration of this process. this also allows us to produce a better population estimate in the end due to the increase of the sample size. ## conclusion in this work, we present a family of methods derived from popular population size estimators that have been widely used in ecology in the past several decades. we show that our family of methods is able to assess language systems under different representations effectively, i.e. using contextualized word and sentence embeddings. our methods show a high correlation to human evaluation on challenging language generation tasks as well as the desired sensitivity to detect mode collapse and quality detriment. in the future, we would like to evaluate our family of metrics on image generation tasks, reinforcing the general applicability of our methods. moreover, we plan to extend our family of methods to also cover popular open populations estimation methods, where the population size may vary over time. in the end, we hope that combining the information from closed and open population methods will improve the overall assessment of language systems, further fostering the adoption of ecology methods in nlp.