Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance (2020)

Abstract: Recent advances in automatic evaluation metrics for text have shown that deep contextualized word representations, such as those generated by BERT encoders, are helpful for designing metrics that correlate well with human judgements. At the same time, it has been argued that contextualized word representations exhibit sub-optimal statistical properties for encoding the true similarity between words or sentences. In this paper, we present two techniques for improving encoding representations for similarity metrics: a batch-mean centering strategy that improves statistical properties; and a computationally efficient tempered Word Mover Distance, for better fusion of the information in the contextualized word representations. We conduct numerical experiments that demonstrate the robustness of our techniques, reporting results over various BERT-backbone learned metrics and achieving state of the art correlation with human ratings on several benchmarks.

## introduction automatic evaluation metrics play an important role in comparing candidate sentences generated by machines against human references. firstgeneration metrics such as bleu @xcite and rouge @xcite use predefined handcrafted rules to measure surface similarity between sentences and have no ability, or very little ability @xcite , to go beyond word surface level. to address this problem, later work @xcite utilize static embedding techniques such as word2vec @xcite and glove @xcite to represent the words in sentences as vectors in a low-dimensional continuous space, so that word-to-word correlation can be measured by their cosine similarity. however, static embeddings cannot capture the rich syntactic, semantic, and pragmatic aspects of word usage across sentences and paragraphs. modern deep learning models based on the transformer @xcite utilize a multilayered self-attention structure that encodes not only a global representation of each word (a word embedding), but also its contextualized information within the context considered. such contextualized word representations have yielded significant improvements on various tasks, including machine translation @xcite , nlu tasks @xcite @xcite , summarization @xcite , and automatic evaluation metrics @xcite @xcite . in this paper, we investigate how to better use bert-based contextualized embeddings in order to arrive at effective evaluation metrics for generated text. we formalize a unified family of text similarity metrics, which operate either at the word/token or sentence level, and show how a number of existing embedding-based similarity metrics belong to this family. in this context, we present a tempered word mover distance (twmd) formulation by utilizing the sinkhorn distance @xcite , which adds an entropy regularizer to the objective of wmd @xcite . compared to wmd, our twmd formulation allows for a more efficient optimization using the iterative sinkhorn algorithm @xcite . although in theory the sinkhorn algorithm may require a number of iterations to converge, we find that a single iteration is sufficient and surprisingly effective for twmd. moreover, we follow @xcite and carefully analyze the similarity between contextualized word representations along the different layers of a bert model. we posit three properties that multi-layered contextualized word representations should have (section 5): (1) zero expected similarity between random words, (2) decreasing out-of-context self-similarity, and (3) increasing in-context similarity between words. as already shown by @xcite , cosine similarity between bert word-embeddings does not satisfy some of these properties. to address these issues, we design and analyze several centering techniques and find one that satisfies the three properties above. the usefulness of the centering technique and twmd formulation is validated by our empirical studies over several well-known benchmarks, where we obtain significant numerical improvements and sota correlations with human ratings. ## related work recent work on learned automatic evaluation metrics leverage pretrained contextualized embeddings by building on top of bert @xcite or variant @xcite representations. sentencebert @xcite ) uses cosine similarity of two mean-pooled sentence embedding from the top layer of bert. bertscore @xcite computes the similarity of two sentences as a sum of cosine similarities between maximum-matching tokens embeddings. mover-score @xcite measures word distance using bert embeddings and computes the word mover distance (wmd) @xcite from the word distribution of the system text to that of the human reference. in the next section we propose an abstract framework of embedding-based similarity metrics and show that it contains the metrics mentioned above. we then extend this family of metrics with our own improved evaluation metric. ## a family of similarity metrics we consider a family of normalized similarity metrics for both word-level and sentence-level representations parameterized by a function c, as follows: for word similarity, x represents a single word vector. a standard choice is defining c(x 1 , x 2 ) = x 1 , x 2 , the inner product between the two vectors. the resulting word similarity metric sim( for sentence similarity, we use @xmath0, x 2 , . . . , x l to denote a d × l matrix composed by l word vectors belonging to the sentence embedded in a d-dimensional space. in what follows, we briefly review existing sentence similarity metrics and show that they belong to our family of similarity metrics eq.( 1 ) with different choices of c(x 1 , x 2 ) (with l 1 and l 2 denoting the sentence length for x 1 and x 2 , respectively). note that we do not consider word re-weighting schemes (e.g. by idf as in @xcite ) in this paper, as their contribution does not appear to be consistent over various tasks. in addition, we assume that all word vectors are already pre-normalized. sentence-bert sentence-bert @xcite uses the cosine-similarity between two mean-pooling sentence embeddings. this is the same as eq.( 1 ) when wordset-cka wordset-cka @xcite uses the centered kernel alignment between the two sentences represented as word sets, where here we assume each word embedding x is precentered by the mean of its own dimensions. we refer to this centering method as dimension-mean centering. moverscore moverscore @xcite measures the sentence similarity using the word mover distance @xcite from the word distribution of the hypothesis to that of the gold reference: the original moverscore does not normalize c(x 1 , x 2 ) by c(x 1 , x 1 )c(x 2 , x 2 ). in practice, we find the performance to be similar with or without such normalization. bertscore bertscore @xcite introduces three metrics corresponding to recall, precision, and f1 score. we focus the discussion here on bertscore-recall, as it performs most consistently across all tasks (see discussions of the precision and f1 scores in appendix c). bertscore-recall uses the sum of cosine similarities between maximum-matching tokens embeddings: for bertscore, since the words are prenormalized, we have c(x 1 , x 1 ) = c(x 2 , x 2 ) = 1 and therefore sim(x 1 , x 2 ) = c(x 1 , x 2 ). note that bertscore is closely related to mover-score, since eq.( 3 ) is the solution of the relaxed-wmd @xcite : ## tempered word mover distance word mover distance @xcite used in moverscore @xcite is rooted in the classical optimal transport distance for probability measures and histograms of features. despite its excellent performance and intuitive formulation, its computation involves a linear programming solver whose cost scales as o(l 3 log l) and becomes prohibitive for long sentences or documents with more than a few hundreds of words/tokens. for this reason, @xcite proposed a relaxed-wmd (rwmd) with only one constraint (see eq.( 4 )), which can be evaluated in o(l 2 ). however, rwmd uses the closest distance without considering there may be multiple words transforming to single words. inspired by the sinkhorn distance @xcite ) which smooths the classic optimal transport problem with an entropic regularization term, we introduce the following formulation, which we refer to as tempered-wmd (twmd): the temperature parameter @xmath1. when @xmath2, eq.( 5 ) reduce to the original wmd as in eq.( 2 ). when t is larger, (5) encourages more homogeneous distributions. the added entropy term makes eq.( 5 ) a strictly concave problem, which can be solved using a matrix scaling algorithm with a linear convergence rate. for example, the sinkhorn algorithm @xcite uses the initial condition the computational cost for each iteration is o(l 2 ), which is more efficient than to that of wmd. although in theory this iterative algorithm may require a few of iterations to converge, our experiments show that a single iteration (i.e., @xmath3. similarly, a tempered-rwmd (trwmd) can be obtained by adding an entropy term to eq.( 4 ): by taking the derivative of the lagrangian of the above objective, the following closed-form solution is obtained: plugging in the optimal π * ij back into the objective yields the following metric: we note that as @xmath4, t log j exp(f j /t ) → max j (f j ), and therefore eq.( 7 ) reduces to eq.( 3 ). in this context, we posit the following three properties that are desirable for word vector representations in context: besides their intuitive appeal, our empirical results (in section 6) do validate that word representations that obey these properties result in higher performance with respect to modeling similarity. since the original word representations does not satisfy these three properties, we explore three methods for centering the word vectors distribution. consider a corpus c containing m sentences {s i }, each of length n i . each word vector is d-dimensional, w i,@xmath5,j , ..., w (d) i,j ]. we propose three candidate word distribution centering approaches: the second term on the rhs is a scalar, which broadcasts to all dimensions of w i,j . we compare these three centering approaches in figure 2 . due to the layer norm operation in the bert models, the dimension mean is a small constant that has little effect after subtraction, and therefore it fails on properties 1 and 2 above. the sentence mean centering achieves approximately zero baseline (property 1), but it also reduces the intra-sim to approximately zero (failing property 3). this indicates the subtraction of sentence mean removes the common knowledge of the words about the sentence, which can have a detrimental effect on modeling similarity. lastly, corpus-mean centering fulfills all three properties above (fig. 2 , bottom row). in this context, we note that, after applying corpus mean centering, cosine-similarity function is reduced to pearson's correlation. since the computational cost of corpus mean centering can be prohibitive for a large dataset, we consider a batch-mean centering approach, which would be especially useful for fine-tuning tasks. in practice, we find that the values obtained from batch-mean-centered word vectors are very close to those of corpus-mean-centered word vectors. therefore and henceforth, we use batch-mean centering to approximate the effect of corpus-mean centering. finally, it is worth noting that corpus (batch)mean centering has recently been applied in normalizing multilingual representations @xcite . however, we are the first to demonstrate its superiority over various other centering methods in single-language by analyzing the inter-layer representation similarities. ## experiments in order to demonstrate the effectiveness of our newly proposed approaches, we conduct extensive numerical experiments based on two commonlyused benchmarks: semantic textual similarity (sts), and wmt 17-18 metrics shared task. our experiments are designed to answer the following questions: (1) are corpus (batch) centered word vectors better than other centered and un-centered word vectors, across different sentence similarity metrics? (2) how do tempered wmd and rwmd compare to their family-relatives moverscore and bertscore? (3) how do the temperature hyperparameter and the sinkhorn iterations affect the performance, and how sensitive are they? to show that our results are consistent across different bert variants, we analyze our similarity metrics over four backbone models: bertbase-uncased, bert-large-uncased, roberta-base and roberta-large, all obtained from the huggingface * transformers package. @xcite found that the better layers for evaluation metric are usually not the top layer, since the top one is greatly impacted by the pretraining task. in particular, @xcite perform an an extensive layer sweep analysis and report that the better layers were always around layer-10 for the base models, and layer-19 for the large models. therefore, in our * https://huggingface.co/models experiments, we used layer-10 for all base models, and layer-19 for all large models. we also present the results of evaluation metrics using different layers in the appendix a and confirm that our main conclusion is not affected by the choice of layer. ## conclusion designing automatic evaluation metrics for text is a challenging task. recent advances in the field leverage contextualized word representations, which are in turn generated by deep neural network models such as bert and its variants. we present two techniques for improving such similarity metrics: a batch-mean centering strategy for word representations which addresses the statistical biases within deep contextualized word representations, and a computationally efficient tempered word mover distance. numerical experiments conducted using representations obtained from a range of bert-like models confirm that our proposed metric consistently improves the correlation with human judgements.