Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in I ndonesia (2025)

Abstract: While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.

## introduction the evaluation of large language models (llms) has shifted from traditional natural language processing (nlp) tasks @xcite to more complex, knowledgeintensive, and reasoning-based challenges. one of the key datasets used to assess these abilities is the massive multitask language understanding (mmlu) @xcite . initially introduced in english, mmlu datasets have also been developed in other languages, including indonesian @xcite , chinese @xcite , and arabic @xcite . these datasets consist of school exam questions across various subjects 1 data can be accessed at https://huggingface.co/ datasets/indolem/indocareer . and education levels, tailored to local curricula. 2 however, they primarily focus on academic subjects, often overlooking vocational and professional expertise, which are more relevant to real-world applications. due to the recent widespread adoption of llms across various domains, including health @xcite , education @xcite , and finance @xcite , evaluating a model's knowledge across professional fields has become crucial. for instance, in healthcare, the model must adhere to ethical standards @xcite and possess expertise in prevalent regional diseases. we should not trust ai-based health recommendations from models that have not passed a competency exam. similarly, in education, the model needs to understand and align with local government teaching guidelines. despite the importance of certification exams in professional fields, such exams have been largely excluded from prior work @xcite . in this paper, we introduce indocareer, a dataset comprising 8,834 multiple-choice questions collected from various indonesian competency exams, certification exams, and vocational school exams. our focus on indonesian addresses the limitations of prior work @xcite and aims to enrich language diversity and local context nuances in nlp datasets, which are predominantly english-centric @xcite . figure 1 shows the distribution of indocareer, which covers 22 different professions across 6 categories: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. additionally, we demonstrate that indocareer is generally robust to option shuffling @xcite when using the entire dataset, but it specifically introduces instability in insurance and finance professions. benchmarks for evaluating language models nusacrowd @xcite represents a significant effort to consolidate scattered datasets for indonesian nlp. while most highquality datasets focus on classical nlp tasks such as sentiment analysis, summarization, and text classification, benchmarks for knowledge-intensive and reasoning tasks have been notably limited until very recently. the introduction of indommlu @xcite , copal-id @xcite , and indoculture @xcite ) marks a step forward in this direction. copal-id and indoculture focus on cultural commonsense reasoning, while indommlu evaluates exam questions across different education levels in indonesia, from primary to high school. despite recent advancements, a significant gap remains in evaluating llms on professional tasks in the indonesian context, as indommlu does not include questions from professional exams. this limitation is not unique to indonesia; professional exam coverage is also limited in similar benchmarks for other languages. for example, english mmlu @xcite and chinese mmlu @xcite include professional exam questions in only 20% of their datasets, while arabic mmlu @xcite has an even lower coverage of just 4%. as llms are increasingly applied across various domains @xcite , there is a pressing need for a benchmark that evaluates their readiness for professional job sectors. indocareer addresses this gap, offering a comprehensive benchmark of professional exams spanning 22 professions, making it the first of its kind in indonesia. ## indocareer indocareer comprises 8,834 multiple-choice questions compiled from indonesian competency exams, certification exams, and vocational school exams across 22 professions. in indonesia, competency exams are commonly required in healthcare professions by the government. certification exams, on the other hand, focus on specific skills within a profession, such as tax accounting in finance. at the high school level, vocational schools offer specialized training in areas like tourism, culinary arts, and fashion design. in figure 1 , ## conclusion we introduce indocareer as the most comprehensive dataset of professional exams across various job sectors in indonesia. the dataset encompasses 22 professions, categorized into healthcare, insurance and finance, creative and design, tourism and hospitality, education and training, and law. evaluations across different llms show that most offthe-shelf models demonstrate vocational and professional expertise below the passing scores. we believe indocareer will be valuable in supporting llm adaptation for various job sectors in indonesia.