M ulti S kill: Evaluating Large Multimodal Models for Fine-grained Alignment Skills (2024)

Abstract: We propose MultiSkill, an evaluation protocol that assesses large multimodal models (LMMs) across multiple fine-grained skills for alignment with human values. Recent LMMs have shown various intriguing abilities, such as solving graph theory problems and explaining visual jokes. However, existing multimodal benchmarks have mainly focused on coarse-grained evaluation (e.g., accuracy), without considering the skill composition required by specific instructions. To this end, we present MultiSkill, designed to decompose coarse-level scoring to a fine-grained skill set-level scoring tailored to each instruction. MultiSkill defines five core vision-language capabilities and divides into 12 skills that are necessary to align with user instructions. For evaluation metrics on specific skills, we propose an LMM-based evaluator for open-ended outputs. Based on the diverse instructions collected from 66 datasets spanning 10 domains, we compare multiple representative open-source and proprietary LMMs and find a high correlation between model-based and human-based evaluations. Our experiments underscore the importance of fine-grained evaluation in providing a holistic view of model performance and enhancing the reliability of the evaluation.

## introduction large language models (llms) have demonstrated remarkable capabilities to follow user instructions by aligning with human values, such as being helpful, honest, and harmless @xcite @xcite . while large multimodal models (lmms), by extending llms with additional modalities such as images, have shown intriguing ability to solve complicated multimodal tasks @xcite ; liu et al., 2023, 2024a), the focus on ensuring their alignment with diverse user instructions remains relatively unexplored @xcite . recent evaluation of lmms relies on independent benchmarks using automatic metrics or overall scoring based on human or llm-based preference @xcite @xcite . however, such evaluation settings are insufficient for three reasons: (1) coarse-grained evaluation: solving complex user instructions usually require integrating different core capabilities, which makes measurement with a single metric insufficient @xcite . as shown in figure 1 , simply assigning a single score showing right or wrong does not tell the whole story, because there could be multiple axes to evaluate the response, such as perception, reasoning, harmlessness, etc. to address the above limitations discussed above, we introduce multiskill (fine-grained large multimodal model evaluation based on multiple alignment skills), an evaluation protocol that employs fine-grained scoring criteria to comprehend lmms from various perspectives, enabling taskagnostic skill evaluation aligned with the provided instructions. building on prior work in skill categorization @xcite , we define 5 vision-language primary abilities, which are divided into 12 fine-grained skills for comprehensive lmm evaluation: logical thinking (logical correctness, logical robustness), background knowledge (factuality, commonsense understanding), problem handling (comprehension, completeness), user alignment (conciseness, readability, harmlessness), and perception (coarsegrained recognition, fine-grained recognition, ocr). first, we collect a total of 962 diverse evaluation instances from 66 multimodal datasets, and annotate the relevant skills necessary for solving the task, domains, and the difficulty level for each instance. then we drop the instances which do not necessarily require perception capability, since in some examples of multimodal benchmarks, the answers can be directly inferred from the textual questions @xcite . next, evaluators assign scores ranging from 1 to 5 for each annotated skill, based on the reference answer and skillspecific scoring rubrics, where the evaluators could be human evaluators or state-of-the-art lmms. we compare and analyze 12 open-source and proprietary lmms on multiskill. we conduct both human-based and model-based evaluations, and observe that their results are highly correlated. our experimental results show that applying finegrained evaluations enhances both interpretability and reliability, increasing the alignment between human and model evaluations. through extensive analysis based on automatic lmm-based evaluation on multiskill, we present several findings: the main contributions of our work are as follows: in terms of evaluating open-ended lmm outputs, motivated by the explorations of llm-based evaluator in the field of natural language processing @xcite , some multimodal benchmarks (such as mmbench @xcite , touchstone @xcite and mm-vet @xcite ) also employ llm-based evaluation. they use advanced llms to compare the model response with reference ground truth answer. this approach encounters significant limitations due to the inherent inability of pure language models to perceive visual contexts directly. in this work, we adopt gpt-4o (openai, 2024), a recently-released state-of-the-art lmm, directly as a judge, and find its high correlation with human annotations. ## multiskill we introduce multiskill, a fine-grained skillbased evaluation protocol designed to assess the alignment of large multimodal models (lmm) with user instructions. first, we define 5 primary abilities, subdivided into 12 distinct skills, which are essential for effectively following user instructions (section 3.1). then, we introduce the construction of the evaluation dataset (section 3.2) and outline the evaluation procedure (section 3.3). note that the evaluation could be conducted by human evaluators or state-of-the-art lmms. finally we discuss the reliability of multiskill and experimentally show the high correlation between human and model-based scores. ## analysis based on automatic evaluation of multiskill while both human-based and model-based evaluations offer reliable and comprehensive analysis, human-based evaluations are time-intensive and costly @xcite . given the high correlation with human-based evaluations shown in ## conclusion in this paper, we introduce multiskill, an evaluation setting for the fine-grained alignment skills of large multimodal models. we categorize a skill taxonomy to evaluate lmms and annotate necessary skills, the target domain, and the difficulty level for each instance. multiskill provides a comprehensive, interpretable and reliable analysis of the capabilities of lmms. also, we observe that applying fine-grained evaluation results in better correlation between human-based and model-based evaluation. we analyze various open-source and proprietary lmms, display the narrowing gap between open-source and proprietary lmms, showing how different underlying language models and tuning data affect the skills of lmms. we expect that multiskill could be utilized for building better lmms and providing meaningful model insights for both developers and practitioners.