CDT : A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and Task (2025)

Abstract: Recent advances in Large Language Models (LLMs) have significantly enhanced their capabilities, highlighting the need for comprehensive evaluation frameworks that extend beyond task-specific benchmarks.However, existing benchmarks often focus on isolated abilities, lacking a holistic framework for assessing LLM capabilities.To address this gap, we propose the \textbf{C} ognition- \textbf{D} omain- \textbf{T} ask (CDT) framework, which comprehensively measures a model’s capabilities across three dimensions.We expand the scope of model capability definitions at the cognitive level by incorporating the Cattell-Horn-Carroll cognitive theory, refining the categorization of model capabilities.We apply CDT in two directions: dataset capability evaluation and data selection. Experiments show that our capability metrics correlate well with downstream performance and can support effective dataset analysis and construction. The experiments on data selection also show significant improvements in both general and specific benchmarks, achieving scores of 44.3 and 45.4, with an increase of 1.6 and 2.2 points over the baselines, respectively. These results validate the effectiveness and practicality of CDT. Source code and models are available at https://github.com/Alessa-mo/CDT .

## introduction recent advances in large language models (llms) have significantly expanded their capabilities. the introduction of reinforcement learning @xcite @xcite and chain-of-thought reasoning @xcite has further enhanced their reasoning abilities. notable llms such as openai's o1 (openai, 2024b) and deepseek r1 (deepseek-ai, 2025) have demonstrated remarkable reasoning capabilities. as llms become more sophisticated, ac-* corresponding author framework open source tagging models multiple dimensions capability decomposition cognition oriented domain oriented task oriented flask fac 2 e instag cdt (ours) however, many of them focus on isolated aspects of model capabilities, such as coding, commonsense reasoning, or specific task performance, and the ability dimensions are always task-oriented and limited, without a holistic framework that systematically categorizes and defines the full spectrum of llm capabilities. for instance, benchmarks like mmlu evaluate knowledge mastery across academic disciplines but overlook dimensions like code generation. recent efforts like flask @xcite and fac 2 e @xcite focus on multi-model comparisons but fall short in capability decomposition and multi-dimensional analysis. additionally, while works like instag @xcite explore capability applications, definitions remain underdeveloped. those works raise the fundamental question: what core capabilities are essential for an effective large language model? to address this, we propose the cognition-domain-task (cdt) framework, a comprehensive multi-dimensional taxonomy for defining, annotating, and utilizing llm capabilities across three dimensions: cognition, domain, and task. our core motivation is that a comprehensive capability analysis must answer three fundamental questions for any given instruction: how to do it, which corresponds to cognition; what it is about, which corresponds to domain; and what to do, which corresponds to task. by deconstructing instructions along these three orthogonal dimensions, cdt provides a holistic and systematic approach to categorizing the full spectrum of llm capabilities. at the cognitive level, cdt incorporates cattell-horn-carroll (chc) theory @xcite , selecting and refining 18 core cognitive abilities suited to llm behavior. at the domain level, we identify nine domain scenarios commonly encountered by llms and further refine these into 33 distinct subdomains. at the task level, drawing inspiration from prior work on dataset construction @xcite @xcite , we systematically categorize task types across diverse instructions, culminating in a taxonomy of 16 task types. we conduct a comparative analysis between existing capability frameworks and our proposed cdt framework, with the results summarized in after constructing the cdt framework, we extend its application to llms in two directions. we first conduct dataset evaluation using two metrics: coverage and balance, which correlate well with downstream performance, demonstrating that cdt can provide practical guidelines for capabilityaware data curation in future dataset construction. then, we apply cdt in data selection to enhance model performance, proposing a diversity-driven selection method to ensure general capability and a capability-oriented strategy which identifies the specific capabilities required by the target test sets. across both general and specific scenarios, experiments show that our data selection methods achieve average scores of 44.3 and 45.4, with an increase of 1.6 and 2.2 points over the baselines, respectively. these results significantly outperform other capability-related methods and baseline approaches. our main contributions are as follows: ## related works definitions of llms' capability research on defining llm capabilities can be grouped into two approaches. the first integrates capabilities with data, optimizing learning through data distribution adjustments @xcite @xcite @xcite . for example, chen et al. (2023) propose a method based on a skill set graph, where mastering one skill aids the acquisition of others, though this method is dataset-specific. similarly, wu et al. (2024) use an mlp-based scoring network for data allocation in fine-tuning, treating datasets as distinct capabilities. the second approach defines model capabilities from task-and domain-specific perspectives, often relying on labeled data for evaluation. zhong et al. (2025) present a hierarchical framework with foundational and complex abilities, while ye et al. (2024) analyze open-source llms to identify four capabilities, subdividing them into 12 skills for comprehensive evaluation. while these approaches offer valuable insights, they often define capabilities in narrow ways, either focusing on isolated aspects, overlooking the underlying cognitive processes, or lacking a holistic, multi-dimensional structure. our work addresses this gap by introducing the cdt framework, which integrates cognitive principles to systematically organize llm capabilities across cognition, domain, and task. applications of llms' capability capability frameworks are often applied to develop evaluation benchmarks for large models. xia et al. (2024) introduce fofo, which evaluates llms' abilities across domains based on format-following. for general evaluation, hendrycks et al. (2021), dubois et al. (2024), and srivastava et al. (2022) have developed broad competency benchmarks. zhong et al. (2025) assess capabilities using prompts, while ye et al. (2024) evaluate models based on responses and instruction alignment. for domain-specific improvements, several studies have proposed different approaches: wang et al. (2024d) integrate capability frameworks with chain-of-thought (cot) to enhance task-specific abilities; lee et al. (2024) introduce thanos for multi-turn dialogue; xu et al. (2023) present lars to improve reasoning by selecting data with similar capabilities; rao et al. (2025) focus on enhancing weak capabilities through error-based learning; and ke et al. (2025) aim to build specialist capabilities by synthesizing high-relevance data from unlabeled text. while prior studies have extended capability frameworks in certain contexts, most still focus on evaluating llm capabilities, with limited exploration of broader applications. to address this, we apply the proposed cdt framework to scenarios such as dataset analysis and data selection, thereby extending its utility beyond conventional evaluation. ## cdt for dataset evaluation and data selection while the cdt framework offers a comprehensive definition of model capabilities, its application to llms remains an area requiring further exploration. leveraging cdt's ability to classify data at the instruction level based on capabilities, we focus on two key application scenarios: evaluating the capability characteristics of existing instruction datasets and guiding the selection of training data to enhance model performance. ## empirical analysis of instruction dataset capabilities datasets we conduct an empirical analysis on eight publicly available instruction datasets widely used in llm instruction tuning. these datasets include: chain of thought @xcite , dolly (conover et al., 2023), open assistant @xcite , flan v2 @xcite , wiz-ardlm @xcite , alpaca-gpt4 @xcite , self-instruct @xcite , and unnatural instructions @xcite . each dataset is annotated using our capability taggers to extract the composite tuples. we then compute the metrics introduced in section 4.1. following @xcite , we collect the alpacaeval @xcite score for model performance. analysis figure 2 visualizes the relationship between coverage, balance, and alpacaeval scores for these datasets. as shown, there is a positive correlation between the two metrics and the model performance. datasets achieving higher scores on both coverage and balance generally yield models with superior alpacaeval scores. notably, topperforming datasets such as wizard, alpaca-gpt4, and open assistant are positioned in the upperright region of the plot, indicating high values for both our proposed metrics. interestingly, open assistant, despite a moderate coverage score, achieves a strong alpacaeval score, potentially due to its exceptional balance score, highlighting the crucial impact of data balance. conversely, datasets like cot and flan, which score lower on these two quality indicators, correspondingly result in models with lower alpacaeval scores. performance variance we also observe that datasets with similar quantitative scores may lead to very different model performance. for example, alpaca, unnatural instructions and self-instruct show similar metrics, but differ widely in effectiveness. we attribute this to the quality of the response annotations: alpaca uses gpt-4-generated outputs, while the latter two rely on earlier models such as text-davinci-002/003, which are substantially weaker. since our capability taggers operate on the instruction side, they may overlook differences in response quality, resulting in the observed discrepancy. we believe this also explains the performance gap between open assistant and dolly. although both are human-annotated, open assistant relies on global crowdsourcing, which results in higherquality responses compared to dolly, whose annotations come from databricks employees. our empirical analysis demonstrates that coverage and balance are effective indicators of dataset quality, and that a combination of comprehensive coverage and well-distributed balance is crucial for training models with high performance. these findings suggest that cdt can serve as practical guidelines for capability-aware data curation in future dataset construction. baselines we conduct the following experiments for comprehensive comparison: configuration we fine-tune the llama2-7b-base model using low-rank adaptation (lora) @xcite , specifically targeting the attention module. distributed training is conducted using deepspeed @xcite . during training, the maximum sequence length is set to 2048, with a batch size of 64 and training epochs as 3. results as shown in ## conclusion in this work, we introduce the cognition-domain-task (cdt) capability framework, offering a comprehensive and systematic approach to classify and decompose the capabilities of llms. by defining cognitive abilities based on cattell-horn-carroll (chc) theory and organizing domain and task capabilities into a structured taxonomy, we enable more nuanced categorization of llm capabilities across various scenarios. additionally, we trained a high-quality annotator on the qwen2.5 model using the cdt framework. we demonstrate the utility of cdt in two key applications. we first apply cdt to dataset evaluation using coverage and balance metrics to assess capability diversity and distribution. we then propose diversity-driven and capability-oriented data selection methods, both of which lead to substantial performance gains across multiple benchmarks. these results confirm the stability and effectiveness of cdt in guiding both dataset evaluation and data selection, highlighting the robustness and practical applicability of the framework.