Layer or Representation Space: What Makes BERT -based Evaluation Metrics Robust? (2022)

Abstract: The evaluation of recent embedding-based evaluation metrics for text generation is primarily based on measuring their correlation with human evaluations on standard benchmarks. However, these benchmarks are mostly from similar domains to those used for pretraining word embeddings. This raises concerns about the (lack of) generalization of embedding-based metrics to new and noisy domains that contain a different vocabulary than the pretraining data. In this paper, we examine the robustness of BERTScore, one of the most popular embedding-based metrics for text generation. We show that (a) an embedding-based metric that has the highest correlation with human evaluations on a standard benchmark can have the lowest correlation if the amount of input noise or unknown tokens increases, (b) taking embeddings from the first layer of pretrained models improves the robustness of all metrics, and (c) the highest robustness is achieved when using character-level embeddings, instead of token-based embeddings, from the first layer of the pretrained model.

## introduction evaluating the quality of generated outputs by natural language generation (nlg) models is a challenging and open problem. human judgments can directly assess the quality of generated texts @xcite . however, human evaluation, either with experts or crowdsourcing, is expensive and time-consuming. therefore, automatic evaluation metrics, which are fast and cheap, are commonly used alternatives for the rapid development of text generation systems @xcite . traditional metrics such as bleu @xcite , meteor @xcite , and rouge @xcite measure n-gram overlap between generated and reference texts. while these metrics are easy to use, they cannot correctly assess generated texts that contain novel words or a rephrasing of the reference text. recent metrics like bertscore @xcite , moverscore @xcite , comet @xcite , bartscore @xcite , and bleurt @xcite adapt pretrained contextualized word embeddings to tackle this issue. these novel metrics have shown higher correlations with human judgments on various tasks and datasets @xcite . however, the correlations are measured on standard benchmarks containing text domains similar to those used for pretraining the embeddings themselves. as a result, it is unclear how reliable these metrics are on domains and datasets containing words outside the vocabulary of the pretraining data. the goal of this paper is to investigate the robustness of embedding-based evaluation metrics on new and noisy domains that contain a higher ratio of unknown tokens compared to standard text domains. 2 we examine the robustness of bertscore, one of the most popular recent metrics for text generation. 3 in order to perform a systematic evaluation on the robustness of bertscore with regard to the ratio of unknown tokens, we use characterbased adversarial attacks @xcite ) that introduce a controlled ratio of new unknown tokens to the input texts. our contributions are: we show that the choice of hidden layer affects the robustness of the evaluation metric. ## bertscore bertscore @xcite computes the pairwise cosine similarity between the reference and hypothesis using contextual embeddings. it forward-passes sentences through a pretrained model, i.e., bert @xcite , and extracts the embedding information from a specific hidden layer. to select the best hidden layer, bertscore uses average pearson correlation with human scores on wmt16 @xcite over five language pairs. for instance, the best layer is the ninth layer for bert base-uncased . ## conclusion embedding-based evaluation metrics will be used across different tasks and datasets that may contain data from very different domains. however, such metrics are only evaluated on standard datasets that contain similar domains as those used for pretraining embeddings. as a result, it is not clear how reliable the results of such evaluation metrics will be on new domains. in this work, we investigate the robustness of embedding-based metrics in the presence of different ratios of unknown tokens. we show that (a) the results of the bertscore using bert-based embeddings is not robust, and its correlation with human evaluations drops significantly as the ratio of unknown tokens increases, and (b) using character-level embeddings from the first layer of byt5 significantly improves the robustness of bertscore and results in reliable results given different ratios of unknown tokens. we encourage the community to use this setting for their embeddingbased evaluations, especially when applying the metrics to less standard domains. in future work, we aim to address other aspects of robustness of evaluation metrics beyond an increased amount of unknown tokens as a result of spelling variation, such as how metrics cope with varying factuality @xcite or with fluency and grammatical acceptability issues @xcite . we also plan to investigate the impact of pixel-based representations @xcite (which are even more lower-level) for enhancing the robustness of evaluation metrics. algorithm 1: count unk token in a bert tokenized sentence 1 def count unk: data: sentence: a tokenized sentence as a list of string output :count: number of unk token of input tokenized sentence 2 count ←-0 3 buffer ←-empty list 4 for token in sentence do 5 if [unk] in token then 6 count ←count +1 7 else if ## in token then 8 add token to buffer 9 else 10 if len(buffer) != 0 then 11 count ←count +1 12 empty buffer 13 end 14 end 15 end 16 if len(sentence) ≥ 2 then 17 if ## in last token of sentence then 18 count ←count +1 19 end 20 end 21 return count