Delving into Evaluation Metrics for Generation: A Thorough Assessment of How Metrics Generalize to Rephrasing Across Languages (2023)

Abstract: Language generation has been an important task in natural language processing (NLP) with increasing variety of applications especially in the recent years. The evaluation of generative language models typically rely on automatic heuristics which search for overlaps over word or phrase level patterns in generated outputs and traditionally some hand-crafted reference sentences in the given language ranging in the forms from sentences to entire documents. Language, on the other hand, is productive by nature, which means the same concept can be expressed potentially in many different lexical or phrasal forms, making the assessment of generated outputs a very difficult one. Many studies have indicated potential hazards related to the prominent choice of heuristics matching generated language to selected references and the limitations raised by this setting in developing robust generative models. This paper undertakes an in-depth analysis of evaluation metrics used for generative models, specifically investigating their responsiveness to various syntactic structures, and how these characteristics vary across languages with different morphosyntactic typologies. Preliminary findings indicate that while certain metrics exhibit robustness in particular linguistic contexts, a discernible variance emerges in their performance across distinct syntactic forms. Through this exploration, we highlight the imperative need for more nuanced and encompassing evaluation strategies in generative models, advocating for metrics that are sensitive to the multifaceted nature of languages.

## introduction in the context of natural language processing (nlp), evaluating generative models typically refers to a two-fold process: while the generated output should first of all be a grammatically and semantically plausible utterance in the target language, it should also fulfil in form or meaning the requirements of a specific task the system is built for. for instance machine translation model output is typically assessed based on how well the system output can represent the meaning of a sentence in another language, while outputs of summarization or question answering systems should be conveying factual information about a given context representing information. the evaluation at hand can then seek to gauge the accuracy, fluency, and appropriateness of the output for the given application at the same time. while a through and accurate evaluation of any nlp system should eventually involve human assessment, due to time and cost considerations, a prominent approach especially during system development typically relies on automatic heuristics which can provide costless reinforcement on the sufficiency or efficacy of the model settings or resources used in system development. automatic evaluation metrics are generally designed with the principle of comparing the similarity of system output to a gold-standard utterance presenting an example of an accurate system output, by relying on the rate of common words @xcite . however, such metrics tend to fall back significantly when the output happens to contain a rephrased version of the context due to stylistic or syntactic variations in the generative process. many languages with rich morphology not only can change in form at the subword level through inflectional or derivational transformations, one can also observe free word order where the same phrase can be written as a combination of the words in many different orders, and still convey the same meaning. in such cases, word-level metrics are known to fail to capture accurate evaluations @xcite @xcite . alternatively, @xcite proposed n-gram match-ing at the character level, which has been more appropriate for the evaluation in morphologically-rich languages. however, matching based approaches still might miss semantic nuances in the generated language. recent studies proposed the alternative approach to use vector similarity in distributed representations @xcite . this method provides a better semantic notion over simple word matching heuristics, yet there is not a well-established understanding on the robustness of pre-trained language representations and how well they may generalize across languages and domains. while valuable, each metric has its challenges, especially given the intricate tapestry of global languages. previous work has compared the performance of evaluation metrics in different tasks @xcite @xcite , however, a task-agnostic analysis that focuses on providing insight on the assessment of generalization capability in generative language models and its measurement across languages with different syntactic typology has never been performed. our study embarks on an extensive examination of evaluation metrics within a linguistic framework where our objective is to understand how these metrics perform in capturing the essence of rephrased language and generalize across diverse syntactic structures and linguistic complexities. for this purpose, we select four prominent automatic evaluation metrics representative of a different approach to evaluation metric formulation: bleu @xcite , chrf @xcite , nist @xcite and bertscore @xcite and use these metrics to compute the similarity across collections of sentences that are paraphrases of each other, in 71 different languages from 12 distinct language families, and measure how different linguistic features affect the applicability of each metric in similarity detection across paraphrased language. our study aims to extend the understanding of evaluation metric performance and highlights potential gaps and areas for further research in considering the future of generative models and how they can be better developed to capture linguistic nuances. through this endeavor, we aim to refine the evaluation process for generative models across multiple languages and promote the study of generative models in potentially many new under-studied languages. ## evaluation metrics for language generation in this study, we focus on sentence-level generation and adopt four commonly used evaluation metrics developed for the automatic evaluation of machine translation. here we briefly define the formulation of each method. ## experimental methodology metrics have undeniably evolved over time, mirroring the advancements in generative models. the above metrics represent this transformation, showcasing the progression from rudimentary n-gram matching to nuanced evaluations via deep learning embeddings. a desired property in each generative language model is to be able to produce plausible language in as many stylistic or syntactic variations the language allows. in order to assess how sensitive each metric is to generalization in the subword or phrase level syntactic structures, i.e. rephrasing, we design a set of experiments that compute similarity between paraphrased utterances in different languages. by the nature of their design, some metrics may be able to capture certain typological forms and patterns better than others, and thus correlate better with languages with those features. in order to test how each metric may suit better capturing grammatical generalization in different languages, we perform an in-depth analysis over the similarity scores and how well they correlate with different types of linguistic features. ## results the metric scores graph (fig. 6 ) presents the distribution of all metric scores computed over paraphrases and organized by language family: on average, we observe the highest bleu scores are computed in the creole language with 0.3692516 points followed by the indo-european language family, with an average of 0.2861689. average bleu scores for japonic, koreanic, niger-congo, afro-asiatic, turkic and uralic languages are much lower with scores ranging as 0.14158, 0.196627, 0.177759, 0.0.23553, 0.22831 and 0.2283065, respectively. for these language groups with relatively complex morphology, we observe the chrf scores, on the other hand, to be much higher on average, with scores of 0.43876 in turkic, 0.50198 in uralic, 0.48934 in afro-asiatic and 0.50487 in niger-congo languages. in japonic, koreanic and sino-tibetan languages, the scores are relatively low, with 0.38995, 0.35384 and 0.30561 respectively, indicating neither n-gram matching based metric are able to capture the rephrasing in example sentences. nist scores are also highest for the indo-european languages with an average of 0.80087 and austroasiatic languages with an average score of 0.70759, however, the scores relatively remain high for morphologically-rich languages, such as in afro-asiatic family, the average nist score is 0.71675, followed by 0.64827 in turkic, 0.69190 in uralic languages, indicating a general improvement for better balancing the more frequent and rare n-gram statistics. in koreanic and niger-congo the scores are very low, with 0.47798 and 0.37228, respectively. finally, the distributed semantic similarity score bertscore obtains the overall, with an average of 0.8684 in indo-european, ranging to slightly different values in different language families with 0.8268 in turkic, 0.85171 in uralic, 0.87922 in afro-asiatic, 0.84763 in niger-congo, 0.87247 in koreanic and 0.86648 in japonic languages, suggesting to be the most applicable metric across languages with varying typological characteristics. we further explore the details of how each metric respond to different linguistic aspects of language by analyzing the correlation between evaluation metric scores and various linguistic typological features. our analysis yields a spectrum of results that underscore the intricacies of language generation evaluation. considering the provided correlation coefficients: ## conclusion this paper provided an analytic study on the evaluation of language generation and how optimal evaluation measures can be developed in a taskagnostic way that can generalize well across different rephrasing choices that are common in natural language. in order to provide insight on the applicability of commonly used evaluation metrics for language generation, we performed extensive experiments on multilingual paraphrase collections and measured the robustness and efficacy of each metric in capturing syntactic variations across languages with varying syntactic typology. our findings confirm the general fallback of surface level matching based heuristics in both applicability and accuracy across languages with different characteristics, and suggest the future of evaluation in lan-guage generation lies in the direction of pre-trained language representation. we hope our study helps better understand how more robust evaluation metrics can be developed, eventually promoting more studies in the development of generative models in many under-studied language families.