E ssay J udge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models (2025)

Abstract: Automated Essay Scoring (AES) plays a crucial role in educational assessment by providing scalable and consistent evaluations of writing tasks. However, traditional AES systems face three major challenges: (i) reliance on handcrafted features that limit generalizability, (ii) difficulty in capturing fine-grained traits like coherence and argumentation, and (iii) inability to handle multimodal contexts. In the era of Multimodal Large Language Models (MLLMs), we propose **EssayJudge**, the **first multimodal benchmark to evaluate AES capabilities across lexical-, sentence-, and discourse-level traits**. By leveraging MLLMs’ strengths in trait-specific scoring and multimodal context understanding, EssayJudge aims to offer precise, context-rich evaluations without manual feature engineering, addressing longstanding AES limitations. Our experiments with 18 representative MLLMs reveal gaps in AES performance compared to human evaluation, particularly in discourse-level traits, highlighting the need for further advancements in MLLM-based AES research. Our dataset and code will be available upon acceptance.

## introduction automated essay scoring (aes) has become an essential tool in educational assessment, providing efficient and consistent scoring for large-scale writing tasks @xcite @xcite @xcite . while aes systems have significantly reduced the workload of human graders, they still face challenges in delivering accurate and detailed evaluations, particularly for trait-specific scoring, which assesses individual aspects of writing quality, such as coherence, creativity, and argumentation @xcite @xcite , 2024). such detailed feedback is critical for guiding students in improving their writing skills, but remains difficult to achieve with existing methods. traditional aes approaches, including statistical models such as support vector machines, rely heavily on handcrafted features such as word frequency and essay length @xcite @xcite . as illustrated in figure 1 (a), they often suffer from ❶ relying on manually engineered features, thus limiting the generalizability across diverse data; ❷ failing to model fine-grained traits such as logical structure and argument persuasiveness; ❸ inability to handle multimodal context, thus struggling to deliver comprehensive and context-aware evaluations @xcite @xcite . the emergence of large language models (llms) and multimodal large language models (mllms) offers a promising solution to these chal-lenges @xcite @xcite . unlike traditional models, llms can capture rich semantic representations directly from text, allowing them to evaluate essays holistically and provide detailed feedback on specific traits @xcite . furthermore, mllms extend this capability by integrating text and image inputs, enabling a deeper understanding of multimodal essay context @xcite . this not only improves scoring accuracy but also addresses the need for nuanced evaluations in more complex writing scenarios. therefore, we propose essayjudge, the first multimodal benchmark for assessing the multigranular aes capabilities of mllms. as shown in figure 1 (b), essayjudge aims to address the aforementioned gaps by ❶ refraining from manually engineered features, using mllms' inherent capabilities to automatically capture complex linguistic patterns and contextual cues, which allows for better generalization across diverse datasets; ❷ leveraging mllms' ability to model multi-granular traits (i.e., lexical-, sentence-, and discourse-levels), enabling more precise and nuanced trait-specific evaluations; ❸ incorporating multimodal inputs including text and image components, enabling mllm to handle complex context, which is crucial for essays on intricate topics. through extensive experiments, we evaluated 18 representative open-source and closed-source mllms, yielding the following key insights: (i) open-source mllms generally perform poorly in aes compared to closed-source mllms, particularly gpt-4o; (ii) closed-source mllms tend to assign lower scores across multiple traits compared to human evaluators, reflecting their stricter scoring criteria; (iii) closed-source mllms perform better in evaluating essays based on single-image setting compared to multi-image one. in general, our findings highlight that there is still a noticeable gap in aes performance compared to human evaluators, particularly in discourse-level traits, underscoring the necessity for further llm research. our contributions can be summarized as follows: by bridging the gaps in existing aes benchmarks, essayjudge contributes to the development of more accurate, robust, and context-aware mllmbased essay scoring systems in the era of agi. ## conclusion in this work, we presented essayjudge, the first multimodal benchmark designed to evaluate the aes capabilities of mllms across lexical, sentence, and discourse-level traits. addressing longstanding limitations in traditional aes approaches, essayjudge leverages mllms' inherent strengths in contextual understanding and multimodal analysis, enabling more precise, traitspecific evaluations without reliance on handcrafted features. our comprehensive evaluation of 18 representative mllms highlights current limitations of mllm-based aes systems. notably, closed-source mllms such as gpt-4 demonstrate superior performance compared to opensource counterparts, yet a significant gap remains in achieving human-level accuracy. we envision that essayjudge will not only drive innovation in aes but also serve as a stepping stone toward broader applications of mllms in educational assessment and beyond. the research community can address the challenges identified and foster the development of more accurate and interpretable aes systems towards agi.