MTVQA : Benchmarking Multilingual Text-Centric Visual Question Answering (2025)

Abstract: Text-Centric Visual Question Answering (TEC-VQA) in its proper format not only facilitates human-machine interaction in text-centric visual environments but also serves as a de facto gold proxy to evaluate AI models in the domain of text-centric scene understanding. Nonetheless, most existing TEC-VQA benchmarks focus on high-resource languages like English and Chinese. Despite pioneering works expanding multilingual QA pairs in non-text-centric VQA datasets through translation engines, the translation-based protocol encounters a substantial “visual-textual misalignment” problem when applied to TEC-VQA. Specifically, it prioritizes the text in question-answer pairs while disregarding the visual text present in images. Moreover, it fails to address complexities related to nuanced meaning, contextual distortion, language bias, and question-type diversity. In this work, we tackle multilingual TEC-VQA by introducing MTVQA, the first benchmark featuring high-quality human expert annotations across 9 diverse languages, consisting of 6,778 question-answer pairs across 2,116 images. Further, by comprehensively evaluating numerous state-of-the-art Multimodal Large Language Models (MLLMs), including Qwen2.5-VL, InternVL-2.5, GPT-4o, GPT-4V, Claude3, and Gemini, on the MTVQA benchmark, it is evident that there is still a large room for performance improvement (InternVL-2.5 scoring 32.2 versus 79.7 for human performance), underscoring the value of MTVQA. By providing a dataset with nuanced multilingual annotations, MTVQA aims to set a new standard for benchmarks, fostering advancements in multilingual visual text comprehension.

## introduction in the era of burgeoning ai, especially in mllms (openai, 2024; team et al., 2023; guo * equal contribution. † corresponding authors. @xcite @xcite @xcite @xcite @xcite @xcite ), text-centric visual question answering (tec-vqa) @xcite @xcite @xcite @xcite @xcite @xcite has served as a de facto gold proxy to evaluate ai models in the domain of text-centric scene understanding. compared with general vqa, tec-vqa places greater emphasis on answering questions that require understanding visual text information within images. it enables individuals without specialized expertise to access applications in text-centric visual environments. however, most advancements in tec-vqa have predominantly concentrated on high-resource languages, e.g., english @xcite @xcite @xcite , chinese @xcite @xcite , japanese @xcite , and etc., thus restricting the applicability of ai models to the global community, particularly populations speaking low-resource languages. to tackle the problem of language diversity, several seminal studies @xcite @xcite in the general vqa field simply leverage translation engines to expand question-answer pairs from highresource to low-resource languages. however, this translation-based approach is not feasible for tec-vqa, as it merely processes text in questionanswer pairs, neglecting the critical visual text needed for answering based on images. although the visual text can be recognized by an ocr (optical character recognition) @xcite @xcite engine and then translated into the target language, this indirect process could lead to a "visual-textual misalignment" problem, due to nuanced meaning, contextual distortion, language bias, and question type diversity. taking the second case in fig. 1 as an example, if either the visual text or the question in russian is recognized or translated problematically, then the question would never be answered correctly. the status quo begs for a question: "can we directly leverage visual text in source language per se for multilingual tec-vqa and what we stand in the mllm era?" in this work, to answer the question above, we establish mtvqa, a novel and high-quality multilingual tec-vqa benchmark, where all images are collected from real-world and meticulously annotated by human experts in nine languages: arabic (ar), korean (ko), japanese (ja), thai (th), vietnamese (vi), russian (ru), french (fr), german (de), and italian (it). more concretely, to ensure the visual-textual alignment at most, the annotation process follows the maker-checker paradigm, where a group of human annotators raises several distinct questions, ranging from simple content extraction to text-related reasoning, and subsequently provides answers. another group then double-checks these qa pairs to ensure accuracy and consistency. consequently, as illustrated in fig. 2 , 6,678 training images and 21,829 questionanswer pairs, as well as 2,116 test images and 6,778 question-answer pairs are obtained, covering more than 20 fine-grained scenarios from both documents and natural scenes, such as menus, logos, maps, bills, ppts, research papers, and etc. to our knowledge, mtvqa is the first tec-vqa dataset to provide native human annotations for multilingual text-rich scenarios, especially for low-resource languages. we further investigate recent representative mllms, including gpt-4o, gpt-4v, gemini, qwen2.5-vl, internvl-2.5, etc., and their performance on mtvqa. as vividly revealed in tab. 2, the general mllm internvl-2.5 is the top performer, followed by qwen2.5-vl and closedsource gpt-4o and claude3 opus. other general mllms like qwenvl max and qwenvl plus show mid-tier performance, while text-centric mllms lag behind. the results unequivocally demonstrate that opportunities for improvement persist within existing mllms when applied in multilingual text-rich scenarios. in summary, the main contributions of this paper can be categorized into four points: benchmark scene manual qa qa language visual text language gpt4v ocrbench multiple text-rich ✓ english english 64.5% textvqa scene text ✓ english english 78.0% docvqa document ✓ english english 88.4% est-vqa scene text ✓ english, chinese english, chinese 72.3% xgqa general 7 languages -67.7% maxm general 7 languages -62.8% mtvqa multiple text-rich ✓ 9 languages 10 languages 22.0% ## mtvqa benchmark the mtvqa benchmark is meticulously established to evaluate the multilingual text comprehension performance of multimodal large language models (mllms). mtvqa covers nine languages: arabic (ar), korean (ko), japanese (ja), thai (th), vietnamese (vi), russian (ru), french (fr), german (de), and italian (it). the data construction involves a comprehensive collection of text-rich images and a two-round human expert annotation process. in terms of data volume, an initial dataset of 8,980 images is compiled. following a data cleaning phase, 8,895 images are retained for annotation. the first round of annotation yielded 8,895 images, corresponding to 28,906 question-answer (qa) pairs. after the sec-ond round of annotation and final quality control measures, the final dataset comprises 8,794 images and 28,607 qa pairs. the benchmark construction cost is divided into two parts: image acquisition and annotation. image acquisition costs about two months and 30,000 dollars. vqa annotation costs about three months and 60,000 dollars. hourly wages vary from country to country, with senior language specialists costing between 20 and 40 per hour and local annotators costing between 8 and 20 per hour. the average time spent by each labeler was roughly 60 days. ## experiments 4.1 baseline models to comprehensively assess mllms' multilingual perception and comprehension capabilities, we select state-of-the-art mllms from three categories: open-source general mllms, open-source textcentric mllms, and closed-source mllms. ## dataset licenses the mtvqa dataset could be used publicly under the cc by-nc 4.0 license. ## acknowledgement this work is supported by the natural science foundation of china (nsfc) no. 62225603. ## conclusion to attack the visual-textual misalignment issue in multilingual tec-vqa, we introduce mtvqa, a new benchmark featuring high-quality human expert annotations in 9 diverse languages. we believe that mtvqa is the first benchmark to provide fully manual annotations tailored to text-centric scenarios. the results obtained from closed-source, general-purpose, and text-centric mllms on our mtvqa dataset indicate that there is still room for improving their performance in multilingual textcentric scenarios. although the mtvqa dataset has limitations, such as the underrepresentation of several lesser-spoken languages, future updates will address these issues by expanding the multilingual scope and including a range of plausible answers. we are confident that this dataset can inspire researchers within the tec-vqa community with new perspectives and ideas.