Incorporating External Knowledge to Enhance Tabular Reasoning (2021)

Abstract: Reasoning about tabular information presents unique challenges to modern NLP approaches which largely rely on pre-trained contextualized embeddings of text. In this paper, we study these challenges through the problem of tabular natural language inference. We propose easy and effective modifications to how information is presented to a model for this task. We show via systematic experiments that these strategies substantially improve tabular inference performance.

## introduction natural language inference (nli) is the task of determining if a hypothesis sentence can be inferred as true, false, or undetermined given a premise sentence @xcite . contextual sentence embeddings such as bert @xcite and roberta @xcite , applied to large datasets such as snli @xcite and multinli @xcite , have led to nearhuman performance of nli systems. in this paper, we study the harder problem of reasoning about tabular premises, as instantiated in datasets such as tabfact @xcite and infotabs @xcite . this problem is similar to standard nli, but the premises are wikipedia tables rather than sentences. models similar to the best ones for the standard nli datasets struggle with tabular inference. using the infotabs dataset as an example, we present a focused study that investigates (a) the poor performance of existing models, (b) connections to information deficiency in the tabular premises, and, (c) simple yet effective mitigations for these problems. we use the table and hypotheses in figure fer to the left column as its keys. in the absence of large labeled corpora, any modeling strategy needs to explicitly address these problems. in this paper, we propose effective approaches for addressing them, and show that they lead to substantial improvements in prediction quality, especially on adversarial test sets. this focused study makes the following contributions: ## challenges and proposed solutions we examine the issues highlighted in §1 and propose simple solutions to mitigate them below. better paragraph representation (bpr): one way to represent the premise table is to use a universal template to convert each row of the table into sentence which serves as input to a bert-style model. @xcite suggest that in a table titled t, a row with key k and value v should be converted to a sentence using the template: "the k of t are v." despite the advantage of simplicity, the approach produces ungrammatical sentences. in our example, the template converts the founded row to the sentence "the founded of new york stock exchange are may 17, 1792; 226 years ago.". we note that keys are associated with values of specific entity types such as money, date, car-dinal, and bool, and the entire table itself has a category. therefore, we propose type-specific templates, instead of using the universal one. 2 in our example, the table category is organization and the key founded has the type date. a better template for this key is "t was k on v", which produces the more grammatical sentence "new york stock exchange was founded on may 17, 1792; 226 years ago.". furthermore, we observe that including the table category information i.e. "new york stock exchange is an organization." helps in better premise context understanding. 3 appendix a provides more such templates. ## experiment and analysis our experiments are designed to study the research question: can today's large pre-trained models exploit the information sources described in §2 to better reason about tabular information? ## comparison with related work recently, there have been many papers which study several nlp tasks on semi-structured tabular data. these include tabular nli and fact verification tasks such as tabfact @xcite , and in-fotabs @xcite , various question answering and semantic parsing tasks @xcite @xcite @xcite , and table-to-text generation and its evaluation (e.g., @xcite . several, models for better representation of tables such as tapas @xcite , tabert @xcite , and tab-struc @xcite were recently proposed. yu et al. (2018, 2021) and eisenschlos et al. (2020) study pre-training for improving tabular inference, similar to our mutlinli pre-training. the proposed modifications in this work are simple and intuitive. yet, existing table reasoning papers have not studied the impact of such input modifications. furthermore, much of the recent work focuses on building sophisticated neural models, without explicit focus on how these models (designed for raw text) adapt to the tabular data. in this work, we argue that instead of relying on the neural network to "magically" work for tabular structures, we should carefully think about the representation of semi-structured data, and the incorporation of both implicit and explicit knowledge into neural models. our work highlights that simple pre-processing steps are important, especially for better generalization, as evident from the significant improvement in performance on adversarial test sets with the same roberta models. we recommend that these pre-processing steps should be standardized across table reasoning tasks. ## conclusion & future work we introduced simple and effective modifications that rely on introducing additional knowledge to improve tabular nli. these modifications governs what information is provided to a tabular nli and how the given information is presented to the model. we presented a case study with the recently published infotabs dataset and showed that our proposed changes lead to significant improvements. furthermore, we also carefully studied the effect of these modifications on the multiple test-sets, and why a certain modification seems to help a particular adversarial set. we believe that our study and proposed solutions will be valuable to researchers working on question answering and generation problems involving both tabular and textual inputs, such as tabular/hybrid question answering and table-to-text generation, especially with difficult or adversarial evaluation. looking ahead, our work can be extended to include explicit knowledge for hypothesis tokens as well. to increase robustness, we can also integrate structural constraints via data augmentation through nli training. moreover, we expect that structural information such as position encoding could also help better represent tables.