Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability (2024)

Abstract: The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area. With the recent emergence of powerful large language models (LLMs), some studies have turned to LLM-based automatic evaluation methods, which demonstrate great potential to become a new evaluation paradigm following traditional string-based and model-based metrics. However, despite the improved performance of existing methods, they still possess some deficiencies, such as dependency on references and limited evaluation flexibility. Therefore, in this paper, we meticulously construct a large-scale NLG evaluation corpus **NLG-Eval** with annotations from both human and GPT-4 to alleviate the lack of relevant data in this field. Furthermore, we propose **Themis**, an LLM dedicated to NLG evaluation, which has been trained with our designed multi-perspective consistency verification and rating-oriented preference alignment methods. Themis can conduct flexible and interpretable evaluations without references, and it exhibits superior evaluation performance on various NLG tasks, simultaneously generalizing well to unseen tasks and surpassing other evaluation models, including GPT-4.

## introduction automated evaluation is crucial for natural language generation tasks, as it measures the performance of related models and consequently promotes the development of nlg research. in the early years, traditional string-based evaluation metrics, such as bleu @xcite , were commonly used. despite their convenience, surface-level matching cannot reliably evaluate texts as they are easily affected by perturbations @xcite , and previous work @xcite ) has indicated their low correlation with human evaluations. with the development of pre-trained language models and related corpora, pre-trained model-based evaluation metrics such as bartscore @xcite and comet @xcite have then been proposed. but their performance remains unsatisfactory compared with human evaluation, and they cannot conduct evaluations on customized aspects, like coherence. recently, large language models (llms) such as chatgpt and llama @xcite have emerged and demonstrated unprecedented performance in instruction following and opendomain generation, which shows great potential for llm-based automated evaluation to become a new paradigm @xcite . existing related studies can be divided into two major categories: directly prompting llms with different optimization methods @xcite @xcite and fine-tuning llms with annotated data @xcite @xcite . however, these approaches still have some limitations: the first category often relies on proprietary llms such as gpt-4, which are high-cost and possibly irreproducible, while the second category tends to have weaknesses of dependence on references, lacking flexibility, or poor interpretability. in this paper, we propose themis, an 8bparameter llm specifically designed and trained for nlg evaluation with more comprehensive capabilities. our themis can evaluate various nlg tasks, including uncommon ones like questionanswering evaluation (versatility), in a referencefree manner (independence). moreover, it allows for specific and customized evaluation aspects and criteria, including overall quality and more finegrained aspects (flexibility), and its evaluation contains corresponding analysis and explanation together with the rating (interpretability), as shown in figure 1 . we believe that an ideal evaluator should be convenient to use and possess these characteristics. the comparison between related methods and themis is shown in to obtain high-quality training data for nlg evaluation, we conducted a comprehensive survey of relevant studies and collected corresponding resources, finally selecting 58 evaluation datasets with human annotations across 9 common nlg tasks. given the importance of evaluation criteria, we meticulously proofread each dataset and manually supplemented the missing descriptions. in addition, gpt-4 is treated as an additional annotator to supplement and validate the collected data while providing evaluation analyses. ultimately, we constructed a large-scale evaluation corpus, nlg-eval, which contains about 0.5 million samples with meta information. it aims to alleviate the issue of scattered and scarce data in the area of nlg evaluation and facilitate relevant research. furthermore, we proposed a multi-perspective consistency verification method to select relatively more reliable data from the constructed nlg-eval corpus. empirical methods are also employed to ensure sufficient diversity and balanced distribution of the data as much as possible, resulting in approximately 67k training samples. moreover, we designed specific preference alignment, guiding the construction and utilization of preference data through evaluation ratings, to improve the evaluation capabilities of the fine-tuned model. experimental results show that our themis achieves better overall evaluation performance over previous evaluation models on common nlg tasks, including summarization, story generation, and so on. we also conducted more in-depth analyses, including generalization tests on unseen tasks like the instruction-following evaluation as well as aspecttargeted perturbation tests to verify the reliability. overall, our main contributions are as follows: method vers. inde. flex. inte. open. ## nlg-eval corpus despite abundant data on nlg tasks, the corresponding high-quality evaluation data remains scarce and scattered due to the high cost of professional human annotations. previous related methods either used a small amount of human annotations to train regression models, such as unieval @xcite and x-eval @xcite , resulting in limited task coverage and a lack of evaluation analyses, or they entirely relied on llms to generate synthetic data, which raised reliability concerns @xcite . to address this challenge, we formally defined the nlg evaluation task and clarified the involved elements. sub- d ia lo g u e r e s p o n s e g e n e ra ti o n sum mar izat ion st or y ge ne ra tio n d a ta to t e x t te xt sim pli fic ati on co nt ro lla bl e ge ne ra tio n g r a m m a t ic a l e r r o r c o r r e c t io n m ac h in e tr an sl at io n paraph rase releva nce spec ificit y kno wle dge use co rre ctn es s co nt ex t ma in te na nc e u nd er st an da bi lty c on te nt r ic hn es s c o n si st e n cy n a tr u a ln e ss o v e ra ll q u a li ty a p p ro p ri a te n e s s c o h e re n c e in te re st in g n e ss f lu e n cy g ra m m at ic al c or re ct ne ss en ga gi ng ne ss co ns ist en cy fa ctu al co ns ist en cy fac tua lity sen time nt con siste ncy faithf ulness specificity coverage fluency selfcohe renc e acc ura cy inf orm ati ve ne ss co he re nc e o ve ra ll q ua lit y r el ev an ce a sp ec t c o ve ra g e r e a d a b ili ty a s p e c t r e le v a n c e e m p a th y c la ri ty r e le v a n c e c oh es iv en es s fl ue nc y co he re nc e gr am m at ica lity co mp lex ity cha ract er dev elop men t enga geme nt likability length overall quality surpr ise rela ted nes s log ica lity in for m at ive ne ss ph ra si ng c o rr e c tn e s s f lu e n cy d at a c ve ra g e st ru ct ur al si m pl ici ty flu en cy ade qua cy simp licity meaning preserva tion grammaticality attribute relevance coher ence flue ncy fac tua l co nsi ste ncy irr ele va nt in for ma tio n co ns ist en cy o ve ra ll q ua lit y m is si ng in fo rm at io n f lu e n cy o v e ra ll q u a li ty m e a n in g p re s e rv a ti o n o v e rc o rr e c ti o n s e m a n ti c s g ra m m a ti ca ll y fl ue nc y lo ca le co nv en tio n te rm in ol og y st yle acc ura cy ove rall qua lity fluenc y semantical appropriateness semantic similarity n a tu ra ln e ss s em an ti c a d eq u ac y r e le v a n c e g ra m m a ti c a li ty t e x t s tr u c tu re se m an tic s ov er all qu ali ty in te re s ti n g n e s s ov er al l qu al ity o ve ra ll q u a lit y sequently, we surveyed a large number of existing related studies and compiled 58 evaluation datasets with human annotations across 9 nlg tasks, totaling 0.5 million samples. they have undergone our meticulous proofreading, and the missing but critical content has been supplemented, such as evaluation criteria. additionally, we utilize the competent gpt-4 for supplementary evaluations, including analyses and ratings. the corpus has also been equipped with meta information, aiming to promote the development of related research. ## methodology existing studies have highlighted that data quality is more crucial than quantity in llm finetuning @xcite . therefore, we adopted empirical methods to improve the diversity and balance of data distribution and designed a multi-perspective consistency verification approach to obtain high-quality samples from the constructed corpus. then they would be used for supervised fine-tuning based on open-source llms. to further enhance the evaluation capability and alignment of the model, we proposed a preference data construction and training method guided by evaluation ratings to improve the fine-tuned model. ## reliability analyses to analyze and investigate the reliability of our themis, we consider two aspects: generalizability, which means the model can effectively evaluate unseen nlg tasks during training, and robustness, which means the model can accurately evaluate texts containing noises and perturbations. ## conclusions in this paper, we propose a large-scale and comprehensive nlg evaluation corpus and an llm dedicated to nlg evaluation, themis. the nlg-eval corpus encompasses 9 common nlg tasks across 58 datasets, comprising about 0.5 million samples with evaluations from both human and gpt-4. based on the constructed corpus, we propose a specialized multi-perspective consistency verification method to select high-quality supervised fine-tuning data, together with preference alignment guided by evaluation ratings, to train our themis. it can be applied to various nlg tasks with flexible and interpretable evaluation in a reference-free manner. extensive experiments demonstrate that our themis performs well on different nlg tasks and can be generalized well to unseen tasks. our model and resources have been released, and we hope they will promote further research in the field of nlg evaluation.