CDS : Data Synthesis Method Guided by Cognitive Diagnosis Theory (2025)

Abstract: Large Language Models (LLMs) have achieved significant advancements, but the increasing complexity of tasks and higher performance demands highlight the need for continuous improvement. Some approaches utilize synthetic data generated by advanced LLMs based on evaluation results to train models. However, conventional evaluation methods fail to provide detailed, fine-grained profiles of LLMs, limiting their guidance for data synthesis. In this paper, we introduce the **Cognitive Diagnostic Synthesis** (CDS) method, which incorporates a diagnostic process inspired by **Cognitive Diagnosis Theory** (CDT) to refine evaluation results and characterize model profiles at the knowledge component level. Based on these diagnostics, we propose two diagnosis-synthesis strategies for weakness-targeted data synthesis. Additionally, we present an enhanced data augmentation and selection pipeline to improve the quality and diversity of synthesized data. Our experiments with several open-source models show significant improvements across multiple benchmarks, achieving up to 6.00% improvement in code generation, 13.10% in mathematical reasoning, and 5.43% in academic exams. Code and data are available on GitHub https://anonymous.4open.science/r/cds-04D1.

## introduction large language models (llms) have demonstrated remarkable capabilities across diverse tasks. however, the increasing complexity of emerging tasks and the limitations revealed in real-world applications highlight the critical need for continuous improvement of llm performance. to achieve continuous improvement, researchers typically analyze the model's evaluation metrics, then refine or supplement the corpora accordingly for subsequent iterations @xcite . for example, if llms are found to perform poorly in mathematical tasks, more math data will be deliberately integrated into the dataset for the next training cycle. in this process, advanced llms (e.g., gpt-4) are increasingly utilized as data synthesizers to automate training data generation and augmentation @xcite @xcite , thereby reducing reliance on costly manual annotation. however, there are two limitations in this process: 1) coarse-grained evaluation. conventional metrics, such as overall accuracy, focus solely on binary (correct/incorrect) outcomes for each test sample, providing a summary of model performance at the dataset level. figure 1 highlights the limitation of these metrics: when a model performs well on most of the knowledge assessed but makes a mistake in percentage calculation, leading to an incorrect final result, conventional metrics would simply classify such cases as "incorrect," failing to identify the weakness in percentage calculation. as a result, the lack of granular evaluation limits the attribution of errors to specific sub-skills or competencies, which are referred to as knowledge components (kcs) in educational theory @xcite , thus hindering the precise identification of weaknesses. 2) the coarse-grained evaluation limits the guidance for subsequent data synthesis, leading to generated data being general and insufficiently targeted at the specific weaknesses of the model. recent studies attempt to use erroneous questions as seed data for synthesis to align the generated data with the model's weaknesses @xcite , but they still treat each error in isolation, failing to map and summarize observed mistakes to underlying capability deficiencies. thus, these methods may correct superficial errors but fail to address fundamental weaknesses at kc level. to address these limitations, we draw inspiration from cognitive diagnosis theory (cdt)-an educational framework that uses "diagnosis" to systematically map assessments to mastery of kcs, identifying specific strengths and weaknesses in students' abilities. in this paper, the llms to be enhanced are treated as "student." we apply this diagnostic approach to summarize their performance in evaluations, profiling their capabilities at the kc level. this profiling then guides advanced llms to synthesize data aimed at improving the weak kcs in the next training cycle. specifically, we introduce the cognitive diagnostic synthesis (cds) method. first, we propose two diagnosis-synthesis strategies from different diagnostic perspectives, using advanced llms as data synthesizers: 1) global strategy: diagnosis at the dataset level with fine-grained metrics such as kc accuracy. these metrics quantify mastery of each kc, helping identify weak mastery kcs and generating tailored training data. 2) fine-grained strategy: diagnosis at the question level, leveraging the analytical capabilities of advanced llms @xcite . we use advanced llms to perform cognitive diagnosis on specific erroneous cases, identifying kcs requiring remediation. these analyses generated during the diagnostic process are then integrated into the synthesis prompt to expand the length of the chain-of-thought (cot), as long cots enhance generation quality @xcite . these synthetic data will undergo augmentation through data rewriting and fusion to enhance their diversity and comprehensiveness. following this, we propose a two-stage data selection process to ensure data quality. in stage 1, an advanced llm is used to filter out erroneous data. in stage 2, a novelty score, cds score , is designed, which references global diagnosis outcomes to select high-quality, weakness-relevant data. our contributions are as follows: ## cds method in the framework shown in figure 2 , we annotate the test samples in the benchmark with their kcs to evaluate the model and diagnose the evaluation results to identify weak kcs, which serve as targets to guide subsequent data synthesis. these synthetic data will undergo augmentation and selection processes to ensure quality, and then be used for supplementary training of the student model. the details are provided as follows. ## conclusion in this paper, we introduce the cognitive diagnostic synthesis (cds) method, inspired by cognitive diagnosis theory, which refines evaluation results and characterizes model profiles at the knowledge component level. building on the knowledge component diagnostic construct, we optimize conventional evaluation metrics, design dual diagnosis-synthesis strategies, and propose a novel data selection method. leveraging advanced llms, we automate the generation of weaknesstargeted, high-quality instructional data. we applied the synthesized data to small llms, such as qwen1.5-7b-chat and llama3-8b-instruct, achieving significant improvements in code generation, mathematical reasoning, and academic testing. notably, cds achieves these improvements without relying on expensive closed-source llms like gpt-4, instead using only the open-source qwen2-72b-in this paper, (1) due to cost limitations, our advanced llm selection is restricted to the opensource qwen-72b-instruct, which is not the most cutting-edge model available. given the current limitations in both the model's analytical and generative capabilities, the full potential of cds remains to be explored. in future work, we plan to experiment with more advanced models, such as @xcite and @xcite .