KIQA : Knowledge-Infused Question Answering Model for Financial Table-Text Data (2022)

Abstract: While entity retrieval models continue to advance their capabilities, our understanding of their wide-ranging applications is limited, especially in domain-specific settings. We highlighted this issue by using recent general-domain entity-linking models, LUKE and GENRE, to inject external knowledge into a question-answering (QA) model for a financial QA task with a hybrid tabular-textual dataset. We found that both models improved the baseline model by 1.57% overall and 8.86% on textual data. Nonetheless, the challenge remains as they still struggle to handle tabular inputs. We subsequently conducted a comprehensive attention-weight analysis, revealing how LUKE utilizes external knowledge supplied by GENRE. The analysis also elaborates how the injection of symbolic knowledge can be helpful and what needs further improvement, paving the way for future research on this challenging QA task and advancing our understanding of how a language model incorporates external knowledge.

## introduction decades of development in question-answering research have seen numerous methods focusing on unstructured text, structured knowledge bases, or semi-structured tables. recent work @xcite has discovered a new challenge in applying these techniques to the financial domain. the study proposed a qa task on financial reports compiled as a tabular and textual dataset for question answering (tat-qa). each question has an associated table and multiple paragraphs, making a hybrid data structure. tat-qa requires a certain level of financial knowledge to extract evidence from tables and texts, making it an appropriate choice for our study. our motivation is to examine whether injecting symbolic knowledge help the model better understand financial concepts. as shown in figure 1 , we can inject the entity information of companies (dbpedia:bce_inc), financial terms (dbpedia:share_repurchase), and common knowledge (dbpedia:europe), among others. the coverage and accuracy of the information depend on the entity linking method. nevertheless, we expect certain common entities to appear in a text-question or table-question pair. we hypothesized that this commonality helps the qa model to focus on the target answer spans, and our analysis provided evidence to confirm the hypothesis. in summary, we introduced the knowledgeinfused question answering (kiqa) model for tabular-textual data. we designed our experiment to evaluate the end-to-end results and investigate the strengths and weaknesses of the injection method to provide insights for future research. our main contributions are as follows: our analysis aims at understanding how language models utilize symbolic knowledge. we intend for this work to stimulate more studies into the mechanism of these models as we advance their capabilities and applications. ## kiqa model kiqa is a qa model built from tagop, a baseline model for the tat-qa dataset, to evaluate symbolic knowledge injection into a qa model for a domain-specific dataset with tabular and textual structure. with the stated objective, we strictly applied the architecture of tagop but replaced the underlying lm, roberta, with luke to obtain knowledge-infused representations. following tagop, kiqa consists of three main components: 1) evidence extraction, 2) reasoning and 3)knowledge injection. ## experiments and results our experimental settings aim to measure the effect of injecting symbolic knowledge into a domainspecific tabular/textual qa model. we chose the financial domain for evaluation since research involving knowledge-infused language models in this domain is still limited. as for the dataset, tat-qa provides extensive and high-quality samples with complex and realistic tabular and textual data. ## analysis we have learned from our experimental results that injecting entity information helped improve the model's performance on textual data. this conclusion seems reasonable given that we did not provide the model with the same information for the tabular input. however, in some cases, the infused text-only entity information negatively affects the model's ability to handle tabular input. our analysis attempts to answer the following questions: ## discussion the qa model used the infused knowledge to focus on the more relevant information (q1). however, only 25.20 % of the answers are in the paragraphs, explaining the limited improvement. we did not anticipate the margin to be substantial since luke's em score on the development set of the squad 1.1 dataset @xcite was only 1.01 % (88.9 → 89.8). injecting entity information to luke resulted in 0.21 % improvement (94.8 → 95.0). however, since our baseline score is much lower, it was reasonable to expect a higher increase (@xmath0: 8.86 % for single text spans). our analysis revealed that the irrelevant entity information interfered with the model's decision, which is why the knowledge-infused models underperformed the baseline model (q2). there is still a gap in tat-qa's tabular data where genre did not perform well, requiring further study involving entity-linking models specialized in tabular data. solving the problem of unrelated entity information interfering with the model's prediction is also another challenge. ## conclusion we investigated the effect of external knowledge infusion on a hybrid tabular/textual qa model in the financial domain. the results indicated an improvement, especially to the textual part of the data. our attention-weight analysis shows the model's ability to utilize the injected knowledge and reveals the challenges involving the hybrid structure of the data. as a result, this study has paved the way for future research to incorporate entity-linking models specialized in tabular data and find a solution that enables the model to integrate tabular and textual symbolic knowledge more efficiently.