MULTITAT : Benchmarking Multilingual Table-and-Text Question Answering (2025)

Abstract: Question answering on the hybrid context of tables and text (TATQA) is a critical task, with broad applications in data-intensive domains. However, existing TATQA datasets are limited to English, leading to several drawbacks: (i) They overlook the challenges of multilingual TAT-QA and cannot assess model performance in the multilingual setting. (ii) They do not reflect real-world multilingual scenarios where tables and texts frequently appear in non-English languages. To address the limitations, we propose the first multilingual TATQA dataset (MULTITAT). Specifically, we sample data from 3 mainstream TATQA datasets and translate it into 10 diverse languages. To align the model TATQA capabilities in English with other languages, we develop a baseline, Ours. Experimental results reveal that the performance on non-English data in MULTITAT drops by an average of 19.4% compared to English, proving the necessity of MULTITAT. We further analyze the reasons for this performance gap. Furthermore, Ours outperforms other baselines by an average of 3.3, demonstrating its effectiveness.

## introduction question answering over the hybrid context of tabular and textual data (tatqa) is an important task @xcite , which is widely used in dataintensive fields, such as finance and science, gaining increasing attention @xcite . enhancing the tatqa capabilities of models can significantly aid in extracting useful information from hybrid data. the heterogeneous evidence brings challenges to the tatqa task since it requires the model to link the relevant information in the table or text according to the entities in the question @xcite @xcite . english 󰏅 text @xcite : arm tormenta ( a-302 ) is a missile boat ... its sister ship is arm huracán . table question what is the sister ship of the ship sold to mexico in 2004? predicted answer arm huracán name ⋯ fate ins romah (halberd) ⋯ active ins geula (salvation) refitted and sold to mexico in 2004 as arm tormenta [8] ins keshet (bow) active ⋯ chinese 󰎩 text [8]：arm tormenta ( a-302 )是一艘导弹 艇 ... 其姐妹舰是arm huracán。 table question 在2004年卖给墨西哥的船的姊妹船是什 么？ predicted answer ins geula (salvation) 名称 ⋯ 命运 ins romah (halberd) ⋯ 现役 ins geula (salvation) 改装后于2004年出售 给墨西哥，改名为arm tormenta [8] ins keshet (bow) 现役 ⋯ figure 1: comparison of the english and chinese examples in multitat. entities with the same color annotation represent corresponding entity information. in chinese, the richness of lexical expressions makes it more challenging for the model to link relevant information, leading to the incorrect predicted answer. to evaluate the model capabilities on the tatqa task, several datasets are proposed @xcite @xcite . for example, hybridqa @xcite , tat-qa @xcite , and @xcite respectively construct english tatqa datasets in the domains of wikipedia, finance, and science. however, these datasets focus solely on english, having the following shortcomings: (i) they cannot adequately assess the tatqa performance in the multilingual setting, overlooking the challenges of multilingual tatqa. as shown in figure 1 , the complex lexical expressions of different languages pose challenges for models to link information across hybrid contexts @xcite . (ii) they create a gap with real-world multilingual scenarios, as domains such as finance and science contain substantial amounts of non-english tables and text @xcite @xcite . to address the limitations, we propose the first multilingual tatqa benchmark, comprising parallel data in 11 diverse languages. first, we introduce the multilingual tatqa dataset (multitat). to ensure the high quality of multitat, we sample data from three mainstream english tatqa datasets and employ a combination of machine translation and manual revision to translate them into 10 languages. in total, multitat consists of 250 questions from 233 hybrid contexts, covering three domains: wikipedia, finance, and science. to enhance the performance of multitat on non-english languages, we propose a baseline to bridge the performance gap between english and non-english on tatqa (ours). to align the model tatqa capabilities in english with other languages, especially low-resource languages, ours is divided into two modules: linking non-english information and reasoning in english. specifically, ours first identifies relevant information from tables and text according to the entities in the question through linking and then uses this information to perform reasoning in english by generating programs. we evaluate the performance of ours, compared with a series of baselines on multitat. experimental results indicate that the performance of non-english languages drops by an average of 19.4% compared to english on all baselines, highlighting the necessity of multitat. ours outperforms other baselines by an average of 3.3, demonstrating the effectiveness. analysis experiments reveal that the tatqa capabilities across languages are not only influenced by resource availability but also by their specific linguistic characteristics. error analysis shows that the performance decline in non-english tatqa is primarily due to the reduced ability to link relevant information, apply formulas, and follow instructions. our contributions are as follows: ## multitat the input of multitat consists of a question, the hybrid context including the table and text, and the output is the answer. additionally, we annotate the rationale, which is the reasoning process of answering the question. we refer to each question, along with its table, text, rationale, and answer, as an instance. for each instance, we annotate 11 diverse languages. we first describe the construction process of multitat, which combines automatic translation with manual error correction, following previous works @xcite @xcite , as shown in figure 2 . ## conclusion to address the limitations of the existing qa datasets on the hybrid context of tabular and text data (tatqa), we introduce the first multilingual tat-qa dataset multitat. specifically, we sample data from mainstream tat-qa datasets, and translate it into 10 diverse languages. to enhance the tatqa performance in non-english languages, we propose a baseline (ours). ours links the relevant information from the hybrid context and reasons in english. we conduct a series of experiments and observe a 19.4% performance drop for non-english languages compared to english. error analysis reveals that this decline is primarily due to the increased difficulty in linking relevant information in non-english texts and the reduced ability to apply formulas and follow the instructions. furthermore, ours achieves an average improvement of 3.3 over other baselines, demonstrating its effectiveness. analysis suggests that the performance of tatqa across languages is influenced not only by high-resource versus low-resource languages but also by the inherent characteristics of the language. xuanliang zhang, dingzirui wang, baoxin wang, longxu dou, xinyuan lu, keyan xu, dayong wu, qingfu zhu, and wanxiang che. 2024a. scitat: a question answering benchmark for scientific tables and text covering diverse reasoning types. preprint, arxiv:2412.11757. yidan zhang, boyi deng, yu wan, baosong yang, haoran wei, fei huang, bowen yu, junyang lin, fei huang, and jingren zhou. 2024b. p-mmeval: a parallel multilingual multitask benchmark for consistent evaluation of llms. preprint, arxiv:2411.09116. yusen zhang, jun wang, zhiguo wang, and rui zhang. 2023. xsemplr: cross-lingual semantic parsing in multiple natural languages and meaning representations. in proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 15918-15947, toronto, canada. association for computational linguistics. yilun zhao, hongjun liu, yitao long, rui zhang, chen zhao, and arman cohan. 2024a. financemath: knowledge-intensive math reasoning in finance domains. in proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 12841-12858, bangkok, thailand. association for computational linguistics. yilun zhao, yitao long, hongjun liu, ryo kamoi, linyong nan, lyuhao chen, yixin liu, xiangru tang, rui zhang, and arman cohan. 2024b. docmath-eval: evaluating math reasoning capabilities of llms in understanding long and specialized documents. in proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 16103-16120, bangkok, thailand. association for computational linguistics. fengbin zhu, wenqiang lei, youcheng huang, chao wang, shuo zhang, jiancheng lv, fuli feng, and tat-seng chua. 2021. tat-qa: a question answering benchmark on a hybrid of tabular and textual content in finance. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3277-3287, online. association for computational linguistics. fengbin zhu, ziyang liu, fuli feng, chao wang, moxin li, and tat seng chua. 2024. tat-llm: a specialized language model for discrete reasoning over financial tabular and textual data. in proceedings of the 5th acm international conference on ai in finance, icaif '24, page 310-318, new york, ny, usa. association for computing machinery.