2 C olumns1 R ow: A R ussian Benchmark for Textual and Multimodal Table Understanding and Reasoning (2025)

Abstract: Table understanding is a crucial task in document processing and is commonly encountered in practical applications. We introduce 2Columns1Row, the first open-source benchmark for the table question answering task in Russian. This benchmark evaluates the ability of models to reason about the relationships between rows and columns in tables, employing both textual and multimodal inputs. 2Columns1Row consists of six datasets, 28,800 tables, that vary in the complexity of the text within the table contents and the consistency of the values in the cells. We evaluate the models using text-only and multimodal approaches and analyze their performance. Through extensive evaluation, we demonstrate the limitations of current multimodal models on this task and prove the feasibility of a dynamic text-based system utilizing our benchmark. Our results highlight significant opportunities for advancing table understanding and reasoning, providing a solid foundation for future research in this domain.

## introduction document processing has emerged as an essential component in various production scenarios, enabling automated extraction, understanding, and analysis of information from different types of documents. a key challenge in this field is understanding tables, often addressed through table question answering (tableqa) @xcite . tableqa involves interpreting tabular data and answering questions based on that information, requiring a good grasp of both the table structure and its content. large language models (llms) have significantly advanced natural language processing (nlp) by demonstrating strong generalization across diverse tasks. a critical application involves table analysis, where tables are typically serialized into textual formats for llm processing. recent approaches leverage large vision-language models (lvlms), combining visual and textual representations to better capture tabular structure and semantics @xcite . despite these advancements, state-of-the-art lvlms still underperform on complex table-related tasks @xcite . furthermore, the lack of publicly available benchmarks for intricate tables, notably for non-english languages, inhibits progress in developing specialized models for this domain. to address these issues, we present 2columns1row, a detailed benchmark for tableqa in the russian language. 2columns1row consists of six datasets that vary in complexity based on the text within the table contents and the consistency of values in the cells, totaling 28,800 instances. we evaluated the performance of several llms on 2columns1row and closely examined their errors, identifying specific patterns in their behavior, especially when dealing with more complex tables. our results highlight the challenges even the most advanced llms face in table analysis. additionally, we assessed the dynamism of the benchmark to ensure its consistency when reassembled. additionally, we investigated the effects of various prompts, table formats, and fine-tuning on the performance of llms. the contributions of the paper are as follows: tasks related to table processing are prevalent in real-world scenarios @xcite , both in production settings and academic research. an application of machine learning is enhancing the automation of the table handling process and extracting valuable insights. however, the difficulty lies in the fact that plain text is used during pre-training of neural language models, which generally lacks the specific structure inherent in tables. to address this, techniques have been developed for adjusting models for tabular data using position embeddings, various attention mechanisms, and learning objectives @xcite @xcite . in recent times, llms have been developing rapidly and demonstrating impressive results in various areas, including the challenges of table understanding, such as tableqa @xcite . due to the versatility of llms, the use of llmspecific techniques remains relevant, including instruction-tuning @xcite , in-context learning @xcite , chain-of-thought (cot) reasoning processes @xcite , and even the use of autonomous agents @xcite , which are becoming increasingly popular. some approaches fine-tune llms, for example, structlm @xcite and tablellm @xcite , which enhance the comprehension of table structures and facilitate complex reasoning for advanced analysis. the rapid development of llms necessitates the creation of suitable benchmarks for a comprehensive evaluation of these models' capabilities and their comparison. nevertheless, the existing benchmarks based on table processing (pasupat and liang, 2015) were mostly constructed for the english language. moreover, there are only a few complex benchmarks for the russian language @xcite and none with table semantic comprehension. to evaluate the abilities of modern llms in table analysis in russian, we present 2columns1row, an extensive and complex synthetic benchmark that incorporates diverse datasets and frequently realworld task formulations for table understanding, effectively addressing the limitations of existing benchmarks. ## experiments we have conducted numerous experiments in textonly and multimodal setups using both open-source and proprietary llms. we employ the official api for all proprietary models (gigachat-2-family llms; gpt-4o) and deepseek-v3 (for optimization purposes). for other models, we accessed them through a vllm library-based server on a set of 8 nvidia a100 gpus. to provide a deterministic and accurate model response for all gigachat-2 models, we used the following settings for generation: @xmath0, @xmath1; for other models, including both text-only and multimodal, we applied @xmath2. we randomly chose five questions for each dataset and table size in all experiments. we selected the query column evenly from all columns, except for the target column, which was always excluded. ## conclusion we present 2columns1row, the first open-source benchmark for tableqa in russian, which covers the model's ability to reason about the relationships between rows and columns in a table using both textual and multimodal modalities. this benchmark offers a comprehensive and dynamic tool for evaluating and improving model performance, thereby advancing the field of intelligent document processing. it assesses textual and multimodal models across diverse tables, demonstrating the viability of a dynamic text-based system for table understanding. the findings highlight significant opportunities for enhancing table understanding and reasoning, establishing a strong foundation for future research in this critical area of document processing. and scale, it may limit the linguistic diversity and complexity of queries. importantly, the generation process incorporates a uniform prior; it does not inherently favor or "teach to" any specific class of models, ensuring a fair and unbiased evaluation framework. a key direction for future work is to incorporate complex, real-world datasets to better reflect the challenges of practical applications and to enhance the naturalness and difficulty of the queries. furthermore, developing a dynamic benchmark structure is crucial for mitigating data contamination and leakage issues in future evaluations.