Diagnosing Failures in Large Language Models’ Answers: Integrating Error Attribution into Evaluation Framework (2025)

Abstract: With the widespread application of Large Language Models (LLMs) in various tasks, the mainstream LLM platforms generate massive user-model interactions daily. In order to efficiently analyze the performance of models and diagnose failures in their answers, it is essential to develop an automated framework to systematically categorize and attribute errors. However, existing evaluation models lack error attribution capability. In this work, we establish a comprehensive Misattribution Framework with 6 primary and 15 secondary categories to facilitate in-depth analysis. Based on this framework, we present AttriData, a dataset specifically designed for error attribution, encompassing misattribution, along with the corresponding scores and feedback. We also propose MisAttributionLLM, a fine-tuned model on AttriData, which is the first general-purpose judge model capable of simultaneously generating score, misattribution, and feedback. Extensive experiments and analyses are conducted to confirm the effectiveness and robustness of our proposed method.

## introduction with the rapid development of large language models(llms), assessing the performance of llms has become an essential research topic @xcite @xcite . a solid evaluation method is capable of providing highquality opinions to guide the llm in its continuous improvement @xcite . the application of llm-as-a-judge model @xcite has drawn significant attention because of its potential to rival human assessment. access to high-performing large language models such as gpt-4 @xcite , 2023) is generally limited to the openai api due to their proprietary. considering the need to avoid potential risks of commercial apis like high cost and data leakage, researchers have commenced training their own judge models @xcite @xcite . for instance, @xcite proposes prometheus, an open-source language model designed to induce evaluation with feedback, which provides a detailed explanation for why a given answer would be awarded a specific score. however, both the open-source and closedsource llms mentioned above concentrate solely on the score, and feedback during the evaluation and neglect to identify the types of errors made in these response errors, as shown in figure 1 . online llm platforms such as chatgpt 2 , doubao 3 , and hunyuan 4 , generate vast quantities of user queries and model responses daily, and manually reviewing each error in this enormous volume of data is im-practical. therefore, platforms require a systematic method to efficiently trace errors back to their root causes-a process termed error attribution. the accurate error attribution enables rapid diagnosis of core issues, informing targeted improvements across critical development phases: pre-training and post-training. in addition, relying on manual annotation to classify errors often leads to inconsistencies due to subjective interpretations and varying evaluation criteria. this lack of standardization affects both efficiency and accuracy. in contrast, a structured framework for error analysis ensures clearer categorization and attribution. this systematic process allows developers to prioritize targeted improvements, accelerating model optimization, and enhancing overall performance, which leads to stronger model robustness, better user experiences, and higher satisfaction. therefore, systematic classification of errors, known as error attribution @xcite , is essential to diagnose weaknesses of llm and guide iterative refinement. to this end, we propose misattributionllm, a 7b llm with error attribution capability that is not only equipped to score the llms responses and generate appropriate feedback but also able to provide detailed misattribution. we establish a comprehensive misattribution framework with 6 categories at the first level and 15 at the second level, to facilitate subsequent analysis and improvement of llms. based on misattribution framework, we present attridata, a high-quality dataset that is manually annotated and crafted to encompass a variety of comprehensive evaluation tasks, representing realistic user demands. by fine-tuning qwen2.5-7b(abbreviated from qwen2.5-7b-base) @xcite with the attridata, we obtain the misattributionllm. on the attridata test dataset, misattributionllm achieves a pearson correlation of 0.935 with human evaluators, which is higher than gpt-4 (0.817), and significantly exceeds gpt-3.5 (0.411). in terms of the performance of error attribution, misattri-butionllm achieves a micro-f1 score of 0.829, surpassing 7 open-source and closed-source llms. furthermore, when human evaluators are tasked with selecting the higher-quality feedback in pairwise comparisons, misattributionllm is chosen over gpt-4 in 60.41% of the cases and outperforms gpt-3.5 with an 85.20% win rate. to the best of our knowledge, we are the first to introduce misattributionllm, an open-source model that simultaneously generates scores, misattribution, and feedback, enabling comprehensive evaluation of llms. in conclusion, our work delivers three key contributions: evaluation method with the development of large language models (llms), recent studies have employed gpt-4 or fine-tuned llms as judge models @xcite @xcite . for example, @xcite introduces pandalm, a fine-tuned llm designed to assess generated text and provide explanations regarding its reliability across various preference datasets. prometheus @xcite stands out as an open-source llm tailored for fine-grained evaluation, capable of adapting to a wide range of scoring rubrics. moreover, critiquellm @xcite demonstrates the beneficial effects of generated critiques as scalable feedback, enhancing the quality of llm outputs. flask @xcite employs natural language instructions to conduct fine-grained, multidimensional evaluations of generated text. more evaluation methods are detailed in appendix a. error attribution extensive research on error detection has been conducted in natural language processing (nlp). @xcite suggested that criticized the limitations of conventional metrics like bleu @xcite , adopting for error-type analysis as a crucial supplement for comprehensive response quality assessment. tiger-score @xcite ) introduces an evaluation model which utilizes error analysis to aid in scoring and increase interpretability. these advancements underscore the critical role of error analysis in nlp evaluation. @xcite addresses the issue of errors in model responses, but the types of error they identify are limited, and there is a lack of trained judge models to tackle these issues. to address these limitations, we propose a comprehensive misattribution framework and train a judge model capable of identifying and categorizing errors. ## method an overview of our method is illustrated in figure 2 . the process can be generally divided into three main steps: data construction, supervised finetuning, and inference. the misattribution framework is described in detail in section 3.1. in sections 3.2 and 3.3, we present the construction and analysis of attridata. lastly, the fine-tuning procedure for misattributionllm is outlined in section 3.4. ## experiments in this section, we explain our experiment setting, which includes the list of experiments, metrics, and baselines that we used to evaluate the performance of llms. additionally, we present a detailed ablation study analyzing model components in section 4.4, and an in-depth case study in appendix e. ## conclusion in this paper, we construct a comprehensive misattribution framework with 6 primary and 15 secondary categories, designed to facilitate in-depth analysis of llms' performance. based on misattribution framework, we present attridata, a highquality dataset with misattribution alongside score and feedback. we also propose misattribution-llm, an innovative open-source, general-purpose llm with the capability of error attribution, which is capable of error attribution and specifically designed for fine-grained evaluation. extensive experiments and analyses are conducted to confirm the effectiveness and robustness of our proposed method. we believe that our work will contribute to advancing the evaluation and analysis of llms.