Automatic Text Evaluation through the Lens of W asserstein Barycenters (2021)

Abstract: A new metric BaryScore to evaluate text generation based on deep contextualized embeddings ( e.g. , BERT, Roberta, ELMo) is introduced. This metric is motivated by a new framework relying on optimal transport tools, i.e. , Wasserstein distance and barycenter. By modelling the layer output of deep contextualized embeddings as a probability distribution rather than by a vector embedding; this framework provides a natural way to aggregate the different outputs through the Wasserstein space topology. In addition, it provides theoretical grounds to our metric and offers an alternative to available solutions ( e.g. , MoverScore and BertScore). Numerical evaluation is performed on four different tasks: machine translation, summarization, data2text generation and image captioning. Our results show that BaryScore outperforms other BERT based metrics and exhibits more consistent behaviour in particular for text summarization.

## introduction automatic evaluation (ae) of natural language generation (nlg) is a key problem towards better systems @xcite . it allows to assess the quality of generated text without relying on human evaluation campaigns that are expensive and time consuming @xcite . for instance, it becomes crucial to design automatic and effective metrics with simultaneous goals: (i) to be able to compare, to control and to debug systems without relying on human annotators @xcite ; and (ii) to improve the learning phase of models by deriving losses that are better surrogate of human judgment than the widely used cross-entropy loss @xcite . a plethora of automatic metrics has been introduced these last few years and may be grouped into two general classes: trained @xcite @xcite and untrained metrics @xcite . in this paper, we mainly focus on untrained metrics that can be further split into three subgroups: string matching @xcite @xcite @xcite , edit based @xcite @xcite and embedding based metrics @xcite @xcite . both string matching and edit based metrics fail to assign reliable scores when reference and candidate convey the same meaning with distinct surface forms @xcite (e.g., case of synonyms and paraphrases). these shortcomings have been addressed by metrics based on continuous representations. recently, they have benefited from contextual embeddings such as elmo @xcite and bert @xcite . perhaps, the most known are bertscore @xcite , moverscore @xcite or sentence mover @xcite which optimise word mover distance (wmd) @xcite foot_0 , a particular instance of optimal transport (ot) problem. originally introduced by @xcite the wmd is used to compute the wasserstein distance between text documents relying on a single layer embedding such as glove @xcite or wor2vect @xcite . to apply wmd with multi layers embedding several recipes have been proposed. bertscore selects the best layer based on a validation set. however, the selection of the validation set is arbitrary while on the other hand, a single layer selection does not exploit the information available in other layers @xcite @xcite . moverscore and sentence mover attempt to leverage the information available in other layers by aggregating the layers using a power mean @xcite . in addition to adding extra hyper-parameters, this aggregation method relies on euclidean topology which induces a geometrical discrepancy as the final cost is computed using a wasserstein distance. our contributions. we introduce baryscore a novel metric, which addresses aforementioned aggregation pitfalls by relying on wasserstein barycenters and evaluate its performance on four different tasks: neural machine translation, text summarization, image captioning and data2text generation. our main contributions can be summarized as follow: ## related work the goal of nlg is to generate coherent, readable and informative text from some input data (e.g., texts, images and tables). however, the exact definition of each of these three criteria remains task-dependent and thus, making it hard to provide a unique metric for all tasks. as an example, nmt focuses on fluency, fidelity and adaquatie (hovy, 2 bary stands for barycenter. 1999; @xcite in contrast to summarization where annotators have to focus on coherence, content, readability, grammatically, coherence and conciseness @xcite . in the following, we describe for each of the four considered tasks (i.e., nmt, text summarization, image captioning and data2text generation) the most used metrics. metrics for nmt. most of the metrics commonly used in nmt rely on comparing surface form (e.g., word, subword, n-gram overlap and edit based distances @xcite ) between text and candidates. perhaps the most popular metrics are the ones used for wmt shared tasks @xcite @xcite which include sentbleu, bleu @xcite , character @xcite , comet @xcite , yisi @xcite , mee @xcite , eed @xcite , chrf @xcite , esim @xcite , prism @xcite to only mention a few among others. a new family of metrics based on pretrained transformers (i.e., bertscore, moverscore) has recently emerged with very good performance in nmt, incorporating deeper semantic information through contextualized representations. metrics for summarization. designing better summarization metrics is an active area of research @xcite and many of these metrics can be further optimized to produce better summaries @xcite . popular metrics include machine translation metrics (i.e., chrf, bleu, meteor @xcite @xcite , bertscore, moverscore or sentencemover @xcite ), rouge @xcite or data statistics (e.g., density and compression ratio) @xcite . metrics for data2text. data2text generation aims at generating text from structured data @xcite @xcite . in the present work, we focus on the webnlg 2020 challenge @xcite which ranks the system using five automatic metrics: bleu, meteor, bertscore, ter and chrf++. metrics for image captioning. task specific metrics for image captioning include cider @xcite that rely on n-grams, leic @xcite using scene graph similarity and pretrained metrics such as spice @xcite . in recent work by @xcite , these metrics are compared with nmt specific metrics (e.g., bleu and meteor). ## background on optimal transport the wasserstein distance (i.e., earth mover distance) which arises from the idea of optimal transport provides a way to measure dissimilarities between two probability distributions. due to its appealing geometric properties, it has found many applications in machine learning such as generative models @xcite @xcite , domain adaptation @xcite , clustering @xcite , adversarial examples @xcite , robustness @xcite or nlp @xcite @xcite . first designed as an optimal transport optimization problem, it relies on minimizing a transport cost between points drawn from all possible coupling measures. its ability to take into account the underlying geometry of the space as well as capture information from distributions with non-overlapping supports makes it a powerful alternative to several dissimilarity measures such as the family of f -divergences. wasserstein distance. let m 1 + (r d ) denote the space of all probability distributions defined on r d with @xmath0. the wasserstein distance between two arbitrary measures µ ∈ m 1 + (x ) and ν ∈ m 1 + (y) is defined through the resolution of the monge-kantorovitch mass transportation problem @xcite : euclidian wasserstein figure 1: euclidian (left) and wasserstein (right) interpolation between densities of two gaussian distributions. ## baryscore metric the construction of automatic metrics usually relies on two paradigms depending on the availability of a reference sentence for each candidate @xcite . here and throughout the paper, we assume that at least one reference is available for each candidate. denote by @xmath1, . . . , ω c nc } the candidate and @xmath2, . . . , ω r nr } the reference composed of n c and n r words, respectively. our goal is to design a metric m : (c, r) → m(c, r) ∈ r + such that the closer to zero the better candidate is. algorithm. our metric m, named baryscore, can be summarized in two steps: with μc = 1 nc nc @xmath3. the previous formulation is similar for the reference text, replacing c by r and α by β in notations. the final embedding, denoted by φ, is then considered as the locations of the barycentric measures, i.e., φ(c) = {x c 1 , . . . , x c nc } and φ(r) = {x r 1 , . . . , x r nr }. computing the wasserstein distance. the last step of our approach is then to evaluate discrete measures induced by the final embeddings, i.e., the candidate and reference barycentric measures μc and μr , using the wasserstein distance, leading to the baryscore given by m(c, r) = w(μ c , μr ). this step boils down to computing the wmd and is similar to the final step in mover-score or sentencemover. the entire procedure is summarized in algorithm 1. algorithm 1 baryscore input: @xmath4, . . . , ω c nc }, @xmath5, . . . , ω r nr }, (φ 1 , . . . , φ l ) pre-trained layers from bert or elmo. compute layers embeddings: φ ℓ (c) and φ ℓ (r) for every 1 ≤ ℓ ≤ l. compute measures: {μ c,ℓ , μr,ℓ } l ℓ=1 . compute wasserstein barycenters: parameters of baryscore metric. our metric is dependant on the choice of the continuous representations (e.g., bert, elmo and roberta) and then its performance will be influenced by the choice of the model. as it is common in concurrent work @xcite @xcite all the results in the paper are obtained with one single model: namely bert-base-uncased. additionally, we report results obtained with the bert fine-tuned on nli release in @xcite , this model is referred as baryscore + in the following. in contrast to previous work (e.g sen-tencemover) that integrates a preprocessing step by removing the stopwords based on a static list, we keep all words. also, our framework provides a natural way to exploit all available layers of the model while previous work relies on a specific subset of them (e.g., moverscore and bertscore). we believe this strengthens the robustness of our approach. comparison with the moverscore. following the footstep of @xcite , the moverscore @xcite applied optimal transport to the output of contextualized encoders (ce) such as bert or elmo. precisely, let's assume that a ce is composed of l layers, the moverscore's context representation is given for each word ω j by φ(ω j ) = t (φ 1 (ω j ), . . . , φ l (ω j )), where the transformation t is either power means @xcite or aggregation routines depicted in @xcite ). the score is then defined by the wasserstein distance between the empirical distributions given by φ(c) and φ(r). the main weakness of this approach is the aggregation step. taking into account the role of the underlying geometry of the probability distribution as well as the interpretability of the transportation flow are high benefits of optimal transport. however, performing wasserstein distance after applying power means, i.e., an aggregation in an euclidian space (see e.g., figure 1 ), does not allow a proper evaluation of the geometry induced by the ce layers in the wasserstein space. indeed, moverscore evaluates a distorted geometry inducing wrong interpretability of the transportation flow. the advantage of exploiting wasserstein barycenter over euclidean aggregation relies on rehabilitating this geometry, as shown in section 6. ## experimental settings in this section, we present our evaluation methods as well as the various dataset used to benchmark our metric. extension of notations. in the previous section, we have only considered a candidate and a reference sentence. in order to evaluate and compare different metrics, we need to extend the previous notations to include the system that generates each sentence. to this end, we will assume that we have a dataset: @xmath6, {c s i , h(c s i )} s @xmath7, where c j i is the i-th text generated by the j-th system, h(c j i ) is human score assign to c j i and r i the reference text associated to c j i ; n is the number of available texts; s the number of different systems. ## numerical results in this section, we study the performance of baryscore on the four aforementioned tasks. ## summary and concluding remarks in this paper, we present a metric named baryscore which relies on optimal transport and solves the geometric discrepancies present in existing metrics that use contextualized embedding with wmd. the present work is carried out in the context of nlg but it introduces a generic theoretically-grounded framework that could be extended to other nlp studies. in particular, it illustrates applications of wasserstein barycenters to combine the different views offered by different layers of a deep neural network. specifically, futur work includes testing wasserstein barycenters in a multimodal setting @xcite , for classification (e.g. emotion @xcite , dialog act @xcite @xcite , stance @xcite ) and controlling style in nlg @xcite @xcite .