Trans- KBLSTM : An External Knowledge Enhanced Transformer B i LSTM Model for Tabular Reasoning (2022)

Abstract: Natural language inference on tabular data is a challenging task. Existing approaches lack the world and common sense knowledge required to perform at a human level. While massive amounts of KG data exist, approaches to integrate them with deep learning models to enhance tabular reasoning are uncommon. In this paper, we investigate a new approach using BiLSTMs to incorporate knowledge effectively into language models. Through extensive analysis, we show that our proposed architecture, Trans-KBLSTM improves the benchmark performance on InfoTabS, a tabular NLI dataset.

## introduction understanding tabular or semi-structured knowledge presents a reasoning challenge for modern natural language processing algorithms. recently, @xcite through tabfact and @xcite via infotabs presented this problem as a natural language inference problem (nli, @xcite , where a model is asked to determine whether a hypothesis is entailed or refuted by a premise, or is unrelated to it (c.f. recently, @xcite highlight the significance of adding world knowledge for the tabular inference task (c.f. in this work, we propose a solution to these issues. we drew inspiration from @xcite and utilize relational connections between premise and hypothesis to extract important knowledge relations from conceptnet @xcite and wordnet @xcite ). this enhancement reduces noise in knowledge addition, resulting in improved knowledge extraction. we embed relational terms in sentences using sentence transformers @xcite to encode semantic representations of the relation, comparable to @xcite , culminating in successful knowledge addition. finally, for effective knowledge integration, we combine these relational embeddings into a word-level language model, using bilstm @xcite , and backpropagate using our proposed bilstm and transformer architecture together to enhance model inferencing capabilities. our proposed model, trans-kblstm, outperforms the earlier baseline, i.e., kg explicit in full as well as limited supervision setting, substantially for some specific categories. furthermore, knowledge addition via trans-kblstm improve model lexical, multi-row and numerical reasoning. we also performed a detailed ablation study to understand the importance of each component. our contributions are as follows: 1. we address the challenges inherent in existing techniques, e.g., kg explicit, for explicit knowledge addition in tabular reasoning. 2. we investigate a more efficient knowledge extraction method that involves using knowledge embeddings rather than directly appending them to the input. 3. we propose a novel architecture, namely trans-kblstm, for integrating word-level knowledge effectively with bilstm's encoders with state-of-the-art transformers such as bert. 4. through extensive experiments, analysis and ablation studies, we demonstrate that trans-kblstm improves reasoning for infotabs dataset. the dataset, and associated scripts, are available at https://trans-kblstm.github.io/ . ## proposed trans-kblstm model we highlight the main model components and their implementation details in this section. we begin with a description of the knowledge relations retrieval technique, followed by a discussion of the model architecture's core components. ## experiment and analysis our experiments study the following questions. rq1: is our proposed model competent in using external knowledge sources effectively to enhance performance across infotabs evaluations sets? rq2: how effective is our approach in settings with little supervision? how much supervision is necessary to outperform benchmark models? rq3: (a) which reasoning types is our proposed model most effective at boosting? (b) is our approach equally effective across all domains, that is, across all table categories? (c.f. §c) ## comparison with related work recently, several papers have been published focusing on nlp tasks involving semi-structured tabular data. examples include tabular nli @xcite , and fact verification @xcite ; @xcite ; @xcite ). the use of external knowledge into tabular data was first explored by @xcite through kg-explicit model described in §3.1. we aim to improve on this benchmark through this extensive study. knowledge integration. traditional approaches to integrating external knowledge into deep learning models do not use contextual embeddings from pre-trained language models. the knowledgebased inference model (kim) @xcite incorporates lexical relations (such as antonyms and synonyms) into the premise and hypothesis representations using attention and composition units. @xcite provides a method to mine and exploit commonsense knowledge by defining inference rules between elements under different kinds of commonsense relations, with an inference cost for each rule. kg-augmented entailment system (kes) @xcite augments the nli model with external knowledge encoded using graph convolutional networks. conseqnet @xcite concatenates the output of the text-based model and the graph-based model and then feeds it to a classifier. @xcite uses lstms and a novel knowledge-aware graph network module named kagnet to achieve stateof-the-art performance on commonsenseqa. bi-cam @xcite ) models incorporate knowledge from conceptnet and aristotuple kgs @xcite by factorized bilinear pooling to improve performance on nli datasets. incorporating external knowledge into language models has been extensively explored in recent times. approaches similar to the tok-ktrans baseline described in §3.1 where external knowledge is added at input level were explored in @xcite ; @xcite . at the representational level, the model understands these external knowledge additions and interacts with these representations using multi-head attention modules @xcite . other approaches include, pretraining on external knowledge corpus to inject knowledge @xcite @xcite , better knowledge representations @xcite , modifications to multi-head attention in pre-trained language models @xcite , designing relation-aware tasks @xcite and integration of knowledge through multi-head attention @xcite . closely related work. @xcite finds that when explicit knowledge is added in the form of word-pair information, models such as @xcite improve performance. however, such models necessitate the use of classic seq2seq architectures such as bilstm to integrate word-level knowledge. in our proposed approach, external knowledge is separately added to the premise and hypothesis using a multi-head attention dot product. to encode the contextual relationships between premise and hypothesis, we use a pre-trained language model, roberta @xcite . we combine the lm embeddings @xcite and bilstm embeddings using a skip connection which preserves the premise-hypothesis relational context and integrates knowledge effectively. ## conclusion and future work in this paper, we introduce trans-kblstm, a novel architecture to integrate external knowledge into tabular nli models. trans-kblstm is shown to improve reasoning on the infotabs dataset. the performance advantage is particularly pronounced in low-data regimes. the reasoning study demonstrates that the model enhances lexical, numerical, and multiple-row reasoning. ablation experiments demonstrate the critical nature of each component in the model's design. we believe that our findings will be valuable to researchers working on the integration of external knowledge into deep learning architectures. performance of the proposed architecture on more datasets can be explored in future studies. looking forward, the application of this architecture to other nlp tasks that can benefit from external knowledge enhanced relational connections between sentence pairs, such as question answering and dialogue understanding.