Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLM s (2025)

Abstract: We introduce MMCRICBENCH-3K, a benchmark for Visual Question Answering (VQA) on cricket scorecards, designed to evaluate large vision-language models (LVLMs) on complex numerical and cross-lingual reasoning over semi-structured tabular images. MMCRICBENCH-3K comprises 1,463 synthetically generated scorecard images from ODI, T20, and Test formats, accompanied by 1,500 English QA pairs. It includes two subsets: MMCRICBENCH-E-1.5K, featuring English scorecards, and MMCRICBENCH-H1.5K, containing visually similar Hindi scorecards, with all questions and answers kept in English to enable controlled cross-script evaluation. The task demands reasoning over structured numerical data, multi-image context, and implicit domain knowledge. Empirical results show that even state-of-the-art LVLMs, such as GPT-4o and Qwen2.5VL, struggle on the English subset despite it being their primary training language and exhibit a further drop in performance on the Hindi subset. This reveals key limitations in structure-aware visual text understanding, numerical reasoning, and cross-lingual generalization. The dataset is publicly available via Hugging Face at https://huggingface.co/ datasets/DIALab/MMCricBench, to promote LVLM research in this direction.

## introduction text-centric visual question answering (vqa) has seen considerable progress with benchmarks such as textvqa @xcite , st-vqa @xcite , docvqa @xcite , visualmrc @xcite , and ocr-bench @xcite , which evaluate models on tasks requiring ocr-based understanding and textual reasoning. more recently, tabular vqa datasets like tablevqa-bench (kim * equal contribution. figure 1 : lvlm performance on mmcricbench-e-1.5k (english) and mmcricbench-h-1.5k (hindi) cricket scorecards. while accuracy on english scorecards peaks at 55.1%, performance on visually similar hindi scorecards remains consistently lower, highlighting a persistent gap in cross-lingual structure-aware numerical reasoning over images. @xcite , tabcomp @xcite , and comtqa @xcite have introduced structure-aware challenges focusing on numerical reasoning and table comprehension. however, as summarized in cricket scorecard images, on the other hand, represent a compelling testbed for evaluating such capabilities. these semi-structured layouts combine tabular numeric data (runs, overs, wickets) with implicit contextual information (e.g., which bowler has bowled the most wides in the match? q3 in figure 2 ), sometimes spanning across multiple images. in this work, we introduce mmcricbench-3k, a novel benchmark for visual question answering on cricket score-1 568 q1 : what is colin munro's strike rate? a1 : 142.86 ✅ q2 : how many batsmen have been dismissed for a duck? a2 : 1 ❌ q3 : how many batsmen had a strike rate greater than 70 in the first innings? a3 : 3 ❌ q1 : what is colin munro's strike rate? a1 : 142.86 ✅ q2 :how many batsmen have been dismissed for a duck? a2 : 1 ❌ q3 : how many batsmen had a strike rate greater than 70 in the first innings? a3 : 4 ✅ example predictions by qwen2.5vl-7b on english (left) and hindi (right) scorecards. q1 is a simple retrieval question, correctly answered in both cases. q2 requires structure-aware, domain-specific reasoning, leading to failure in both. q3 reveals a cross-lingual gap answered correctly on the english scorecard but incorrectly on the hindi one, despite identical content. large vision-language models (lvlms) (llava-1.5 @xcite , minigpt4 @xcite , mplug-owl @xcite , qwen-vl @xcite , and internvl2 @xcite ) have become the de facto approaches for visual question answering tasks, including text-aware visual tasks (penamakuri and mishra, 2024a). recent lvlms such as qwen2.5vl @xcite , mplug-docowl2 @xcite , internvl2 @xcite , and textmonkey @xcite have further advanced the bar on text-aware tasks, including vqa, by incorporating text-aware objectives into their pretraining or instruction-tuning stages. while studies exist to show strong performance of these models on english benchmarks, similar studies to understand their robustness across low-resource languages like hindi foot_0 remains unexplored in the literature. to this end, we leverage our mmcricbench-h-1.5k benchmark to understand and evaluate cross-lingual mathematical reasoning abilities of lvlms. our experiments reveal a consistent performance drop when these lvlms are evaluated on mmcricbench-h-1.5k (as illustrated in figure 1 ), highlighting significant shortcomings in structure-aware, cross-lingual, and intensive numerical reasoning. although advanced paradigms like chain-of-thought (cot) prompting improve performance over naive variant, they still fall short compared to their performance on english scorecards. in summary, our contributions are three-fold: (i) we introduce mmcricbench-3k, a novel structure-aware text-centric vqa benchmark to cover for the shortcomings of existing ocr and table-based vqa benchmarks by incorporating cross-lingual, multi-image, structure-aware, and numerically rich reasoning tasks grounded in the domain of cricket analytics. (ii) we comprehensively benchmark a range of leading lvlms (open and closed-source) across different model sizes and show that they struggle on this benchmark, revealing key limitations in structure-aware visual understanding, numerical reasoning, and cross-lingual robustness. (iii) we conduct extensive ablations incorporating specialized components such as optical character recognition (ocr), table structure recognition (tsr), and advanced prompting strategies including chain-of-thought (cot) reasoning. while these methods improve performance, they still fall short compared to the model's strong results on conventional text-centric benchmarks, highlighting the unique difficulty of our task. ## mmcricbench-3k dataset we introduce mmcricbench-3k, a novel dataset designed to study a visual question answering (vqa) task on cricket scorecard images. cricket scorecard images represent unstructured yet complex tabular images. vqa on such scorecards requires structural understanding, numerical data extraction, and implicit contextual reasoning across image(s). to the best of our knowledge, our work is the first principled work on studying vqa over cricket scorecard images. specifically, we present two sub-benchmarks under mmcricbench-3k: mmcricbench-e-1.5k (with english scorecards) and mmcricbench-h-1.5k (with hindi scorecards), with english question-answer annotations. this dataset is aimed at benchmarking the capabilities of large vision-language models (lvlms) in performing cross-lingual deep mathematical reasoning over semi-structured content. mmcricbench-3k consists of cricket scorecards sourced from various international game formats: odi, t20, test match, and popular regional leagues: the big bash league (bbl, australia) and the indian premier league (ipl, india). we provide carefully curated qa annotations to evaluate the numerical comprehension and deep mathematical reasoning abilities of lvlms. next, we explain the dataset curation pipeline. data collection and annotation: we begin to collect data for our benchmark by identifying publicly available datasets and repositories that contain cricket scorecard information. the initial dataset was obtained from kaggle foot_2 , which provides detailed cricket match statistics in csv format. this dataset includes essential match statistics such as runs, wickets, and strike rates across different cricket formats (international game formats and regional leagues). note that the data curated from the above-mentioned source does not contain scorecard images. ## experiments baselines. we chose the vlms from three selection criteria: (a) vlms with no ocr-aware tasks during their pretraining or instruction tuning stages: llava-next @xcite , and (b) vlms based on the size of their parameters: (i) small vlms (svlms) with parameters less than 5b: smolvlm-500m @xcite , qwen2.5vl-2b @xcite vlms (lvlms) with parameters between 5b-14b: internvl2-8b @xcite , qwen2.5vl-7b @xcite , mplug-docowl2 @xcite , (c) x-large vlms with parameters greater than 14b: llama-3.2-v-11b @xcite and (iii) closed-source vlms: gpt-4o (openai, 2024). ## comparison with related work lvlms for vqa over text images: recent advancements of large vision-language models (lvlms) have transformed visual questionanswering (vqa) tasks into gaining impressive zero-shot performance across diverse scenarios @xcite @xcite ) including text-centric vqa. on these lines, docpedia @xcite processes high-resolution inputs without increasing token sequence length. mplug-docowl @xcite , qwen2-vl @xcite , and textmonkey @xcite further leverage publicly available document vqa datasets to boost text performance. extensions of the llava @xcite ) framework such as llavar @xcite , internvl @xcite , @xcite and @xcite have broadened lvlm capabilities in visual text by leveraging both textual content and visual content, thereby setting a new benchmark for text-centric vqa including their knowledge-aware counterparts (e.g. tex-tkvqa @xcite ). despite these significant strides, lvlms fall short in complex tasks like mmcricbench-3k as discussed in section 3. ## conclusion we presented mmcricbench-3k, a novel benchmark for vqa on cricket scorecards that addresses critical gaps in existing datasets by incorporating cross-lingual understanding, multi-image reasoning, and domain-specific numerical analysis. our evaluation across mmcricbench-e-1.5k (english) and mmcricbench-h-1.5k (hindi) scorecards reveals a significant perfor-8 mance disparity among state-of-the-art lvlms. while these models show reasonable proficiency with english scorecards, they struggle substantially with hindi variants despite identical information content. even advanced prompting strategies like cot fail to bridge this performance gap. these findings highlight a critical weakness in cross-lingual visual reasoning capabilities, highlighting the need for more robust models that can effectively process structured numerical data across language boundaries. as ai applications expand globally, addressing these limitations becomes increasingly crucial. mmcricbench-3k provides researchers with a challenging testbed for advancing lvlm capabilities beyond englishcentric contexts, particularly in domains requiring precise analysis of semi-structured information.