INSTRUCTSCORE : Towards Explainable Text Generation Evaluation with Automatic Feedback (2023)

Abstract: Automatically evaluating the quality of language generation is critical. Although recent learned metrics show high correlation with human judgement, these metrics do not provide explicit explanation of their verdict, nor associate the scores with defects in the generated text. To address this limitation, we present INSTRUCTSCORE, a fine-grained explainable evaluation metric for text generation. By harnessing both explicit human instruction and the implicit knowledge of GPT-4, we fine-tune a text evaluation metric based on LLaMA, producing both a score for generated text and a human readable diagnostic report. We evaluate INSTRUCTSCORE on a variety of generation tasks, including translation, captioning, data-to-text, and commonsense generation. Experiments show that our 7B model surpasses all other unsupervised metrics, including those based on 175B GPT-3 and GPT-4. Surprisingly, our INSTRUCTSCORE, even without direct supervision from human-rated data, achieves performance levels on par with state-of-the-art metrics like COMET22, which were fine-tuned on human ratings.

## introduction although large language models (llms) have led to significant progress in various natural language tasks @xcite @xcite , it remains a challenge to automatically evaluate the quality of text generation across versatile tasks. traditional word overlap metrics, such as n-gram matching, bleu @xcite , and chrf @xcite , along with distance-based metrics like ter @xcite do not best align with human experts' judgements @xcite . they primarily focus on surface form differences between reference and candidate texts @xcite . on the other hand, recent learned metrics such as bertscore @xcite , bleurt @xcite , comet @xcite and error type: incorrect translation has stylistic problems severity: major error location: usually when there is takeaway, explanation: the translation uses an awkward phrasing "usually when there is takeaway," instead of "usually, when there's a delivery." score: -5 human i don't understand what those scores mean i will save you. i am intelligent instructscore. sescore @xcite show a higher correlation with humans on text generation tasks. however, all these metrics produce a single numerical score. these learned metrics lack interpretation of predictions nor link the scores with individual defects in the candidate text. how can we devise a fine-grained explanationbased text generation metric capable of pinpointing concrete error locations, identifying error types, assigning severity labels, and justifying the final score-all simultaneously without relying on human-annotated data. in this paper, we propose instructscore, a method to learn an explainable text generation metric without using human annotated ratings. instructscore provides both a numerical score and a natural language error explanation. to this end, we first extract latent evaluation knowledge from an instruction-following model, such as @xcite , to construct a synthetic dataset with a predetermined explanation structure. next, we determine a range of explanation failure modes and devise automated feedback to meta-evaluate error explanations. finally, we further fine-tune instructscore model on self-generated outputs that optimize feedback scores, resulting in diagnostic reports that are better aligned with humans. we have conduct experiments on a variety of text generation tasks: machine translation, table -to-text, image captioning, commonsense genteration, and keyword-to-dialogue generation. our experimental findings show that the unsupervised instructscore outperforms prior strong baselines on all these tasks. it achieves the best results for the unseen keyword-to-dialogue generation task. surprisingly, instructscore surpasses the supervised bleurt in 6 out of 9 directions and closely matches state-of-the-art comet22 in machine translation. furthermore, we identify a range of failure modes and design an automatic pipeline to pinpoint explanation failures. our refinement step improves human score by 13.7%, leading to a more accurate alignment with human judgment. our instructscore enjoys the following advantages: (i) compact yet competitive: in-structscore's 7b version displays strong performance compared to metrics based on closed-source 175b llms. (ii) explainable: instructscore provides natural language explanations to justify numerical scores. (iii) generalizable: the unsupervised training pipeline does not require humanannotations, making it easily adaptable to different domains and tasks. ## related work learned evaluation metrics supervised metrics optimize performance by directly fine-tuning human rating data, such as comet @xcite and bleurt @xcite , as shown by @xcite and @xcite . however, human rating data is often unavailable. unsupervised metrics use different learning objectives or heuristics on embeddings, such as bert for greedy matching and coverage scoring @xcite , or sequence-to-sequence models for probability estimation @xcite . sescore @xcite and sescore2 @xcite ) train a regression model by synthesizing human-like errors from raw text and using either a pre-trained natural language inference or a multilingual masked prediction model to attach error severity score. supervised metrics can arguably attain higher correlations with human judgments @xcite , while unsupervised metrics, such as sescore @xcite and bertscore @xcite , exhibit greater levels of generalization. however, none of these approaches offer an explanation for the resulting scores, rendering the decision-making processes obscure and less trustworthy. in this paper, we generate a diagnostic report to provide detailed explanations to support metric's final decisions. explainable evaluation metric. recent demand for explainability in evaluation metrics has grown significantly. @xcite introduce a multi-dimensional human evaluation (mqm) framework for machine translation, while @xcite investigates key characteristics of explainable metrics. several metrics derived from those frameworks enhance explainability by differentiating error severity @xcite @xcite . other efforts focus on explanatory aspects of text generation metrics, like error locations @xcite and multi-dimensional assessment @xcite . despite progress, explanations remain unclear. researchers also explore llms' potential in evaluation, as demonstrated by @xcite , but suffers from a lack of explanation. @xcite and @xcite find large models like gpt-3.5 on system-level can correlate to humans and generate rationales. however, these generated rationales are free-form and may not necessarily align with human judgements @xcite . in our work, we explicitly refine instructscore to produce explanations that align with human. ## problem definition our goal is to learn an explainable metric model that not only predicts the quality score of candidate text comparing to a reference but also generates a diagnostic report in natural language. specifically, instructscore assesses the quality of x regarding a reference r by generating an informative diagnostic report, which includes the details about error location l, error type t, severity level se, and explanation e that are associated with the identified error. instructscore consists of a score predictor and an explanation generator (exp-generator) which learns a function f : (x, y) → {(l, good: the translation uses an awkward phrasing "usually when there is takeaway," instead of "usually, when there's a delivery." bad: the translation uses " there is takeaway" instead of " there is takeaway," which alters the meaning of the sentence. with n number of errors. however, such human annotated mapping data for most text generation tasks is scarce due to limited human resources and high annotation costs. to this end, we propose a data construction method to automatically generate high-quality pseudo data to learn f . ## the instructscore approach instructscore assesses the quality of generated texts based on an explainable diagnostic report. building upon this report, instructscore provides an intuitive way to comprehend a model's generation capability, resulting in easier comparison among different models. in particular, we begin by extracting concise yet representative explainable knowledge from a large-scale instructionfollowing model, which is then utilized to train our exp-generator. after carefully analyzing the diagnostic reports produced by our exp-generator, we summarize common failure modes in diagnostic report and ask gpt-4 to identify them. then we transform the gpt-4's feedback into alignment scores using our predefined criteria. finally, we select diagnostic reports that have the highest alignment scores, and further finetune our exp-generator on those self-refined outputs. the overall framework is illustrated in figure 2 . the quality score s for each candidate y is determined based on the number of errors and their severity labels in the diagnostic report. minor errors are given a score of -1 and major errors are given a score of -5. these penalties for errors are weighted to calculate the final score. similar to pre-vious practices @xcite , our metric identifies up to five errors per sentence. ## conclusion in this paper, we present a novel framework for explainable text generation evaluation, addressing the existing black-box limitations associated with learned metrics. we define a set of failure modes to regularize the explanations. we empirically demonstrate that instructscore can be generalized to different domains, tasks and evaluation dimensions, achieving the best ranking compared to other general text generation metrics. lastly, our refinement from automatic feedback can further improve human alignment score, precision, and recall, by 13.7%, 11.6%, and 3.2%, respectively, leading to a more accurate alignment with human requirements. we released the instructscore model for public use and open-source the data and codes.