Benchmarking Multimodal Models for U krainian Language Understanding Across Academic and Cultural Domains (2025)

Abstract: While the evaluation of multimodal English-centric models is an active area of research with numerous benchmarks, there is a profound lack of benchmarks or evaluation suites for low- and mid-resource languages. We introduce ZNO-Vision, a comprehensive multimodal Ukrainian-centric benchmark derived from the standardized university entrance examination (ZNO). The benchmark consists of over 4300 expert-crafted questions spanning 12 academic disciplines, including mathematics, physics, chemistry, and humanities. We evaluated the performance of both open-source models and API providers, finding that only a handful of models performed above baseline. Alongside the new benchmark, we performed the first evaluation study of multimodal text generation for the Ukrainian language: we measured caption generation quality on the Multi30K-UK dataset. Lastly, we tested a few models from a cultural perspective on knowledge of national cuisine. We believe our work will advance multimodal generation capabilities for the Ukrainian language and our approach could be useful for other low-resource languages.

## introduction vision-language models (vlms) have expanded llm capabilities into more domains, allowing for models to work with plenty of new tasks such as ocr @xcite , image captioning, visual question answering and many more. while numerous benchmarks @xcite ) evaluate vlms performance across a range of multimodal tasks, these resources primarily serve english-language models, underscoring a critical gap for evaluating vlms in less-resourced languages. this absence is especially pronounced for ukrainian, where multimodal benchmarks are exceedingly scarce. our work addresses this gap by introducing a suite of ukrainian-specific benchmarks and pre-senting benchmarking results for leading @xcite vlms. to estimate academic knowledge, we developed a new benchmark based on the external independent evaluation (zno) -national university entrance and teacher certification exam @xcite , which includes a large selection of questions across various fields, such as chemistry, mathematics, ukrainian language and literature, etc. besides that, we evaluated all models using @xcite , one of the few existing ukrainian multimodal benchmarks. additionally, for the culture test, we developed a new multimodal benchmark, uacuisine, based on 20 popular ukrainian dishes. we believe that our effort would advance the development of vlms applications for the ukrainian language across academic and business sectors worldwide, wherever it's being used. code, evaluation scripts, and datasets are available at this link: https://github.com/lang-uk/ mmzno-benchmark . ## related work recent years have seen significant development in multimodal benchmarks for evaluating vlms. existing benchmarks can be broadly categorized into three groups. general visual understanding benchmarks include vqa @xcite (1m+ question-answer pairs), gqa @xcite (compositional reasoning), and mmmu @xcite (broad domain reasoning). cultural and multilingual benchmarks are represented by @xcite (11 countries), @xcite (30 languages), and maxm @xcite (7 languages). visual reasoning benchmarks feature clevr @xcite (compositional reasoning), a-okvqa @xcite (external knowledge), and visual7w @xcite (semantic understanding). while these benchmarks provide comprehensive evaluation frameworks, they predominantly focus on english language capabilities. recent multilingual benchmarks often rely on translations rather than culturally-grounded content, highlighting a critical gap for evaluating vlms in underrepresented languages like ukrainian. translationbased benchmarks like xgqa @xcite (9,670 questions in 7 languages) often introduce artifacts and fail to capture cultural nuances @xcite . current cultural evaluations are either too limited in scope (culturalvqa: 2,378 questions across 11 countries) or too narrow in focus (world-cuisines: food-specific across 30 languages). analyzing the worldcuisines, we found three critical limitations regarding ukrainian cuisine: (1) representation was restricted solely to location identification tasks without deeper cultural assessment, (2) the selection of dishes failed to capture the breadth of ukrainian culinary traditions, and (3) several dishes were incorrectly categorized as ukrainian while featuring russian-language captions and representing russian cuisine variants. ## experimental setup for each benchmark evaluation, we used their specific metrics with fixed random seeds for python, numpy, and torch. for zno, we used a temperature of 1 and a maximum output tokens equal to 1024; we noticed that proprietary models produced many tokens before generating an answer. for multi30k, we adopted flickr30k @xcite evaluation methodology as presented in the lmms-eval framework for standardized multimodal evaluation @xcite , employing temperature of 0 and a maximum output tokens equal to 64. for the uacuisine benchmark, we employed the same temperature of 1 and a maximum output tokens equal to 512. we evaluated both proprietary and opensource multimodal language models to provide a comprehensive assessment of current capabilities on ukrainian language tasks. besides standard setting, we measured the same question without images provided in the text-only setting to measure contamination. the lowest theoretical baseline evaluation of zno is to select the first choice in each question, getting a 22% accuracy score. as part of a benchmark, the model falls back on a randomly chosen answer from options if it fails to provide an answer. that's why, as a baseline, we evaluated all models in a text-only setting without images provided and treated similar scores in both settings as failing to beat a baseline. most text-only evaluations score approximately 34%, values close to what we treat as a baseline. ## results & discussion in as for the uacuisine benchmark, the leaderboard is close to a zno one, except for gemini 2.5 pro, which failed to generate recipes and dish ingredients. as shown in while the former factor is a limitation of our work, the latter is a manifestation of cultural and linguistic bias by the models. instruction-following issues. the most prevalent challenge observed across models was inconsistent instruction following in ukrainian. even high-performing models like gpt-4o and gemini frequently failed to respond in the expected format. a notable case is a meta-llama/llama-3.2-90b-vision-instruct, which, instead of ukrainian letters for answers, responds in english ones. we have observed models replying in a much more verbose way than expected by multi30k, therefore we modified prompts with an extra instruction to reply with a sentence for multi30k, but issue persisted. ## conclusions in this work, we introduced a suite of benchmarks to evaluate vlms in ukrainian, addressing a critical gap in resources for low-and mid-resource languages. zno benchmark enables researchers to estimate model performance objectively for ukrainian multimodal generation using expert-made questions. to the best of our knowledge, there were no prior public evaluations of the multi30k-uk dataset for the caption generation task, which we hope will be useful for other researchers in estimating model proficiency for ukrainian in a multimodal setting. as for uacuisine, we hope to highlight cultural issues in vision-language models with our research, providing a framework to measure them objectively in a ukrainian-specific setting. future research directions should focus on extending benchmarks to include more diverse, language-specific tasks, addressing the culture gap. beyond ukrainian, the methodologies introduced here could serve as a template for advancing multi-modal language modelling in underrepresented languages, enabling more inclusive access to ai instruments.