Bilingual Tabular Inference: A Case Study on I ndic Languages (2022)

Abstract: Existing research on Tabular Natural Language Inference (TNLI) exclusively examines the task in a monolingual setting where the tabular premise and hypothesis are in the same language. However, due to the uneven distribution of text resources on the web across languages, it is common to have the tabular premise in a high resource language and the hypothesis in a low resource language. As a result, we present the challenging task of bilingual Tabular Natural Language Inference (bTNLI), in which the tabular premise and a hypothesis over it are in two separate languages. We construct EI-InfoTabS: an English-Indic bTNLI dataset by translating the textual hypotheses of the English TNLI dataset InfoTabS into eleven major Indian languages. We thoroughly investigate how pre-trained multilingual models learn and perform on EI-InfoTabS. Our study shows that the performance on bTNLI can be close to its monolingual counterpart, with translate-train, translate-test and unified-train being strongly competitive baselines.

## introduction tabular natural language inference (tnli) is the task of classifying whether a textual hypothesis is an entailment, contradiction or a neutral extension of the given tabular premise. the task requires a broad range of reasoning abilities, including but not limited to the ability to make lexical, spatiotemporal, and semantic deductions. recently published datasets, tabfact @xcite and infotabs @xcite , have enabled the examination of the tnli task. moreover, sophisticated models based on deep contextual embeddings like bert @xcite , roberta @xcite , etc. trained under the 101ers, the clash h1: john graham mellor plays less instruments than the number of labels he has worked for. h2: joe strummer changed his surname after he became a guitar player. h3: joe strummer was active in the sports industry for over three decades. h1 hi-trl : jon grāham melar un lebaloan kī sankhyā kī tulanā mean kam vādya bajāte haian jinake lie unhoanne kām kiyā hai h2 hi-trl : jo st . ramar ne ek git . ār vādak banane ke bād apanā upanām badal liyā h3 hi-trl : jo st . ramar tīn dashakoan se khel udyog mean sakriya the figure 1: tabular premise followed by human written hypotheses (h1, h2, h3). h1 is entailed entirely from the premise, h2 is neither entailed nor contradictory, and h3 is contradictory. h1 hi-trl , h2 hi-trl , and h3 hi-trl are the transliterations of hindi translations of the former, released as a part of our ei-infotabs dataset. overall, ei-infotabs dataset and our proposed train-eval strategies enable thorough examination of the challenging task of btnli. furthermore, the former aslo serve as a quality benchmark for evaluating the robustness of multilingual models. the dataset and the associated scripts, are available at https://enindicinfotabs. github.io . ## motivation why tabular nli? tabular data is termed as semi-structured as it is neither truly unstructured 1 joe strummer wikipedia data like raw text, nor is it entirely structured like a database. although semi-structured data is based on a structured scaffold, the content can be freeform text with variable length and type. moreover, unlike a database, there is no homogeneity across various data points in a shared context. such structural ambiguity imposes a significant cognitive load while reasoning about it. however, such data is ubiquitous in the real world (e.g. web pages, fact sheets, information tables) and we frequently make inferences from it. @xcite argue that reasoning about semi-structured data is broadly two-fold in nature. it consists of (a.) linguistic reasoning: a semantic deconstruction of the semi-structured data (b.) , and symbolic reasoning: a symbolic execution on the tabular structure.for instance, h 2 in figure 1 requires linguistic reasoning over the phrase "became a guitar player" from the "occupation", and the "instruments" rows of the concerned table. h 1 requires symbolic reasoning in the form of conditional and arithmetic operations on the "labels" and "instruments" rows. whereas, h 3 requires a combination of the two types of reasoning. such interwoven reasoning criteria makes it challenging to model tabular nli task. why indic languages? indian society is largely multilingual and consists of 122 major and 1599 other languages and dialects spanning 6 language families with over 1.3 billion native speakers foot_0 . out of these, 30 languages have more than 1 million native speakers each and over 1 billion speakers cumulatively foot_1 . moreover, india has the second largest online presence with over 749 million internet users and is expected to grow to over 1.5 billion users by 2040 foot_2 . so, development of competent reasoning models for the indic context is essential. however, due to unfair linguistic bias on the web @xcite , there is a disproportionate distribution of text resources for indian languages. indian languages have a limited number of internet resources. thus, they are often known as low web resource languages (lrl) @xcite . for instance, wikipedia entries in hindi are just 2% of those in english, and wikipedia entries in assamese and oriya are 7 times lesser than those in hindi. this implies that a significant fraction of articles and sometime even complete categories are discussed only in the english language wikipedia @xcite . although, efforts have been made to bridge this gap @xcite , there still exist several limitations: (a.) table extraction for an article across languages is a challenge due to absence of wikipedia page links, their infobox tables or important keys of tables, (b.) even if tabular data exists, infobox tables in indian languages are not updated as regularly as their english equivalents @xcite which leaves us with outdated and untrustworthy tabular data for inference, (c.) and lastly, table translation while maintaining the intent, context, and the same quality of the source english language is difficult. often, accurate translation requires the distinction of a language specific domain expert. due to above reasons, tabular data is mostly absent from indic wikipedia articles. thus, fact verification in a bilingual setting wherein, the premise is in english and the claim/hypothesis is in an indic language, is of great significance. moreover, recent advances in multilingual language models @xcite , datasets @xcite , and translation systems @xcite for indian languages have enabled quality examination of several indic nlu tasks which serves as additional motivation to evaluate the task of btnli for indic languages before other low resource languages. @xcite , an english tnli dataset. infotabs consists of 23,738 pairs of tabular premises and textual hypotheses. the hypotheses are human written short assertions with an accompanying nli label, and the tabular premises are based on 2,540 wikipedia infoboxes from 12 diverse categories. moreover, it consists of additional adversarial test sets apart from α 1 which is the standard test set and is lexically and topically similar to the train setα 2 is the lexically adversarial test set which maintains topical similarity and α 3 is the topically adversarial test set. the dev and test sets (α 1 , α 2 , α 3 ) cumulatively consist of 7200 table-hypothesis pairs equally splits on all four sets. ## experimental pipeline we design our experimental pipeline along the lines of the research question: how well do existing pretrained multlilingual language models perform on the btnli task? in this section, we propose various modeling strategies and examine how they might address the challenges and nuances of the proposed inference btnli task. ## results and analysis in this section, we discuss and analyse the results obtained on conducting the experiments as per the various strategies laid out in §4. we present the results in ## further discussion ei-infotabs is the first tabular nli dataset in the indic context which enables preliminary studies in this field. moreover, it introduces bilinguality for fact verification scenarios which is of huge significance in low resource contexts. it motivates the development of cross-lingual reasoning models, and helps in evaluation of robustness of multilingual models. for instance, our experiments on ei-infotabs clearly indicate that muril is a significantly more robust multilingual model when compared to mbert as it is able to generalize its reasoning ability across all indic languages. although, we explain how machine translation doesn't affect the semantics of the hypotheses, it does come with a few challenges. we identified a few instances wherein the indictrans model translates named entities, instead of transliterating them. this is observed only, but not always, when a named entity has an english dictionary word in it. for instance, "death proof ", name of a movie, gets translated and not transliterated in two out of nine hypotheses containing the phrase. this is mostly observed in the movies category. however, this doesn't affect our reasoning models and they perform on par on this category when compared with roberta large 's performance on infotabs. this is so because such translations when shallow parsed indicate that the translated entity still acts as the noun phrase in the sentence. this helps the translation, though technically imperfect, retain the intended semantic structure. multilingual models. multilingual, and specifically cross-lingual transfer @xcite , has been widely discussed in the context of low resource languages. several datasets @xcite @xcite @xcite , benchmarks and leaderboards @xcite @xcite , and evaluation frameworks @xcite @xcite have emerged which focus entirely on evaluation of multilingual nlu. further, multilingual language models have been developed for (a.) natural language understanding @xcite @xcite @xcite , (b.) and natural language generation @xcite . indic resources. indic nlp, recently, has seen a recent surge in the number of datasets @xcite @xcite @xcite , multilingual models @xcite @xcite , toolkits @xcite @xcite , translation systems @xcite , and dedicated benchmarks for evaluation @xcite . this has enabled the indian nlp research community to construct competent models for a variety of challenging nlp tasks. ## conclusion we motivate and introduce the bilingual tabular nli for fact verification tasks, and release ei-infotabs-a first of its kind tabular nli dataset for making inferences in 11 indic languages over english tabular data. our robust quality estimation experiments show that the machine translated datasets closely preserve the semantics of the source and are fluent. we show that pre-trained multilingual models find this task challenging, however, still perform close to the benchmarks on infotabs with translate-test and translate-train providing good performance. the analysis also shows the similarity of inference capabilities across languages. the dataset offers immense potential as it opens up avenues in (a) multilingual tabular nli, (b) bilingual claim verification, (c) and evaluation of multilingual models.