Towards Evaluation of Language Models with Skill Dimensions: A Case Study on Narrative Question Answering (2025)

Abstract: Large language models have demonstrated varying levels of competence across a range of reasoning tasks, but coarse-grained evaluations often do not reflect their specific strengths and weaknesses, particularly in complex tasks such as Narrative Question Answering. In this paper, we advocate for a multi-dimensional skill-based evaluation that assesses models across distinct core skill dimensions. Our proposed skill-focused evaluation framework offers a granular and more realistic measure of model performance, revealing targeted areas for improvement and guiding future development. Experiments on Narrative Question Answering demonstrate that dimension-level analysis captures the multifaceted nature of the task and informs more effective model evaluation.

## introduction large language models (llms) have achieved impressive results across a variety of reasoning tasks. however, current evaluation practices predominantly rely on coarse-grained evaluation that aggregates performance into a single score that does not reflect the strengths and weaknesses of models across different reasoning skills. this limitation poses a significant challenge: without a detailed understanding of where models excel or struggle on a task, it becomes difficult to identify targeted areas for improvement or to accurately estimate their readiness for real-world applications. earlier works on fine-grained skill evaluation @xcite have focused primarily on challenging skills. while this offers useful insight into models' upper limits, it similarly captures only part of the broader skill landscape, leaving room for more comprehensive approaches that consider the full spectrum of reasoning abilities. @xcite makes progress toward addressing issues in coarse-grained evaluation by selecting and evaluating a subset of fine-grained skills drawn from an existing skill taxonomy @xcite . this approach enhances the granularity of evaluation by focusing on specific, essential top skills. however, it does not attempt to cover the full range of skills that might be involved in answering each question. as a result, the evaluation may overlook other relevant skills, providing only a partial view of model capabilities. to fill this gap, we propose a dimensional skill evaluation framework that systematically assesses model performance across distinct core skill dimensions. while earlier works have focused on defining skill dimensions @xcite @xcite , they have not leveraged these dimensions to structure evaluation in a way that more accurately reflects model capabilities. in contrast, our framework introduces a three-level evaluation approach: (1) skill-level evaluation, (2) dimension-level evaluation, and (3) multi-dimensional evaluation. this structure ensures that each skill and dimension contributes proportionally to the final assessment. by explicitly accounting for both skill-level and dimension-level variation, our framework addresses challenges such as skill imbalance and the diverse range of reasoning abilities required in complex question answering tasks. we validate the effectiveness of this skillfocused framework through evaluation of llms on the narrative question answering task. the results demonstrate that dimensional evaluation captures the multifaceted nature of the task and reveals nuanced insights, such as uneven improvements across skill dimensions when scaling model size, and highlights developmental priorities masked by coarse-grained scores. overall, the proposed framework offers a more granular and actionable measure of llm performance, guiding future model development and deployment for complex reasoning tasks. ## dimensional skill evaluation traditional evaluation in qa often provides a single coarse-grained score that aggregates performance across all of the questions. however, these scores do not account for different skills and skill dimensions, potentially masking a model's strengths and weaknesses in specific skills. to address this, we propose a dimensional skill evaluation framework that assesses the model at three levels: skilllevel, dimension-level, and multi-dimensional evaluations. ## evaluation setup we evaluate the narrative question answering abilities of large language models (llms) on the test set of the fairytaleqa (xu et al., 2022) dataset. the test set contains 1,007 questions. we automatically annotated the questions based on skill taxonomy by @xcite . more details on the annotation process can be found in appendix a. for evaluation, we choose llms with different parameter sizes: 1b, 3b, and 8b instruction-tuned variants of the llama-3 @xcite and 1b, 4b, and 12b instruction-tuned variants of gemma-3 @xcite ) models, as well as the gpt-4o @xcite . we use flow judge v0.1 (flowai, 2024), an llm-as-a-judge model, to evaluate models' predictions. we report the average llm-as-a-judge-based accuracy score of 5 model runs. dimensional skill evaluation is conducted with equal skill and dimension weights. additional details on the evaluation setup can be found in appendix b. ## results and discussion skill-level, dimension-level, multi-dimensional, and coarse-grained evaluations of llms on fairy-taleqa are presented in skills within dimensions provide a good comparison point. analyzing skill-level performance within a dimension, such as skill-level results in the scope dimension section of ## conclusion this study argues that a dimension-focused skill evaluation offers a more accurate and insightful assessment of large language models on complex tasks like narrative question answering. unlike coarse-grained evaluation, this approach uncovers specific strengths and weaknesses across skill dimensions, revealing that model improvements are often uneven and concentrated in certain areas. the findings emphasize the limitations of coarsegrained evaluation and advocate for dimensionlevel analysis to guide model development priorities and better reflect real-world performance readiness.