XI nfo T ab S : Evaluating Multilingual Tabular Natural Language Inference (2022)

Abstract: The ability to reason about tabular or semi-structured knowledge is a fundamental problem for today’s Natural Language Processing (NLP) systems. While significant progress has been achieved in the direction of tabular reasoning, these advances are limited to English due to the absence of multilingual benchmark datasets for semi-structured data. In this paper, we use machine translation methods to construct a multilingual tabular NLI dataset, namely XINFOTABS, which expands the English tabular NLI dataset of INFOTABS to ten diverse languages. We also present several baselines for multilingual tabular reasoning, e.g., machine translation-based methods and cross-lingual. We discover that the XINFOTABS evaluation suite is both practical and challenging. As a result, this dataset will contribute to increased linguistic inclusion in tabular reasoning research and applications.

## introduction natural language inference (nli) on semistructured knowledge like tables is a crucial challenge for existing (nlp) models. recently, two datasets, tabfact @xcite on wikipedia relational tables and infotabs @xcite on wikipedia infoboxes, have been proposed to investigate this problem. among the solutions, contextual models such as bert @xcite and roberta @xcite , when adapted for tabular data, surprisingly achieve remarkable performance. the recent development of multi-lingual extensions of contextualizing models such as mbert @xcite from bert and xlm-roberta @xcite from roberta, has led to substantial interest in the problem of multi-lingual nli and the creation of multi-lingual xnli @xcite and @xcite dataset from english mnli @xcite dataset. however, there is still no equivalent multi-lingual nli dataset for semi-structured tabular data. to fill this gap, we propose xinfotabs, a multi-lingual extension of infotabs dataset. the xinfotabs dataset consists of ten languages, namely english ('en'), german ('de'), french ('fr'), spanish ('es'), afrikaans ('af'), russian ('ru'), chinese ('zh'), korean ('ko'), hindi ('hi') and arabic ('ar'), which belong to seven distinct language families and six unique writing scripts. furthermore, these languages are the majority spoken in all seven continents covering 2.76 billion native speakers in comparison to 360 million english language (infotabs) speakers foot_0 . the intuitive method of constructing xinfotabs, i.e., human-driven manual translation, is too expensive in terms of money and time. alternatively, various stateof-the-art machine translation models, such as mbart50 @xcite , marianmt @xcite , m2m100 @xcite , have greatly enhanced translation quality across a broad variety of languages. furthermore, nli requires simply that the translation models retain the semantics of the premises and hypotheses, which machine translation can deliver @xcite . therefore, we use automatic machine translation models to construct xinfotabs from infotabs. tabular data is far more challenging to translate than semantically complete and grammatical sentences with existing state-of-the-art translation systems. to mitigate this challenge, we propose an efficient, high-quality translation pipeline that utilizes name entity recognition (ner) and table context in the form of category information to convert table cells into structured sentences before translation. we assess the translations via several automatic and human verification methods to ensure quality. our translations were found to be accurate for the majority of languages, with german and arabic having the most and least exact translations, respectively. the dataset and associated scripts, is available at https://xinfotabs.github.io/ . ## why the infotabs dataset? there are only two public datasets, both in english, available for semi-structured tabular reasoning, namely tabfact @xcite and infotabs @xcite . we choose infotabs because it includes multiple adversarial test sets for model evaluation. additionally, the infotabs dataset also includes the neutral label, which is absent in tabfact. the infotabs dataset contains 2,540 tables serving as premise and 23,738 hypothesis sentences along with associated inference labels. the tablesentence pairs are divided into development, and three evaluation sets α 1 , α 2 , and α 3 , each containing 200 unique tables along with nine hypothesis sentences equally distributed among three inference labels (entailment, contradiction, and neutral). α 1 is a conventional evaluation set that is lexically similar to the training data. α 2 has lexically adversarial hypotheses. and α 3 contains domain topics that are not present in the training set. the remaining 1,740 tables with corresponding 16,538 hypotheses serve as a training set. ## table representation machine translation of tabular data is a challenging task. tabular data is semi-structured, nonsentential (ungrammatical), and succinct. the tight form of tabular cells provides inadequate context for today's machine translation models, which are primarily designed to handle sentences. thus, table translation requires additional context and conversion. furthermore, frequently occurring named entities in tables must be transliterated rather than translated. figure 1 shows the table translation pipeline. we describe our approach to context addition and handling of named entities in detail in the following subsections §3.1. ## translation and verification as mentioned previously, we now grasp how to represent a table. consequently, these reformatted tables can now be fed into reliable translation models. to accomplish this, we assess many prominent multilingual (e.g., mbart50 @xcite and m2m100 @xcite ) and bilingual (e.g., marianmt (junczys-dowmunt et al., 2018)) translation models as described below: multilingual models. this category of models used includes widely used machine translation models trained on a large number of languages such as mbart50 @xcite which can perform translation between any two languages from the list of 50 languages and m2m100 @xcite which has 100 training languages. apart from these models, we used google translate 7 to compare against our dataset translation quality. ## experiment and analysis in this section, we study the task of multilingual tabular nli, utilizing our xinfotabs dataset as the benchmark for a variety of multilingual models with multiple training-testing strategies. by doing so, we aim to assess the capacity and cross-lingual transferability of state-of-the-art multilingual models. for the inference task, we linearize the table using the "table as struct"-tabfact described in infotabs. ## discussion and analysis extraction vs. translation. one straightforward idea for constructing the multilingual tabular nli dataset is to extract multilingual tables from wikipedia in the considered languages. however, this strategy fails in practice for several reasons. for starters, not all articles are multilingual. for example, only 750 of the 2540 tables were from articles available in hindi. the existence of the same title articles across several languages does not indicate that the tables are identical. only 500 of the 750 tables with articles in hindi had infoboxes, and most of these tables were considerably different from the english tables. the tables had different numbers of keys and different value information. ## conclusion we built the first multilingual tabular nli dataset, namely xinfotabs, by expanding the infotabs dataset with ten different languages. this is accomplished by our novel machine translation approach for tables, which yields remarkable results in practice. we thoroughly evaluated our translation quality to demonstrate that the dataset meets the acceptable standard. we further examined the performance of multiple multilingual models on three validation sets of varying difficulty, with methods ranging from the basic translation-based technique to more complicated language-specific and intermediate task finetuning. our results demonstrate that, despite the models' success, this dataset remains a difficult challenge for multilingual inference. lastly, we gave a thorough error analysis of the models to comprehend their cross-linguistic transferability, robustness to language change, and coherence with reasoning.