Hypothetical Training for Robust Machine Reading Comprehension of Tabular Context (2023)

Abstract: Machine Reading Comprehension (MRC) models easily learn spurious correlations from complex contexts such as tabular data. Counterfactual training—using the factual and counterfactual data by augmentation—has become a promising solution. However, it is costly to construct faithful counterfactual examples because it is tricky to maintain the consistency and dependency of the tabular data. In this paper, we take a more efficient fashion to ask hypothetical questions like “in which year would the net profit be larger if the revenue in 2019 were $38,298?” , whose effects on the answers are equivalent to those expensive counterfactual tables. We propose a hypothetical training framework that uses paired examples with different hypothetical questions to supervise the direction of model gradient towards the counterfactual answer change. The superior generalization results on tabular MRC datasets, including a newly constructed stress test and MultiHiertt, validate our effectiveness.

## introduction machine reading comprehension @xcite trains deep models to understand the natural language context by answering questions. however, these deep models easily learn spurious correlations (a.k.a. shortcuts) @xcite @xcite between the context and answer, e.g., entries at the first column have higher chance to be chosen as answers in complex financial tables. consequently, the context understanding is incomplete or even biased, leading to significant performance drop on testing examples without such shortcut (e.g., f1score drops from 74.9 to 40.0, cf. counterfactual training @xcite @xcite ., * * corresponding author. in this work, we utilize an economic alternative: asking hypothetical questions (hqs) @xcite ) by imposing the factual example with a counterfactual assumption, without the cost of maintaining the table consistency and dependency. the construction cost of a hypothetical example is undoubtedly lower than the counterfactual example foot_0 . a hypothetical example consists of a hypothetical question and factual context, which has the equivalent effect on the answer to the corresponding "ideal" counterfactual example. as a concrete case in figure 1a , the counterfactual example is derived from the factual example according to the assumption "if the revenue in 2019 were $38,298", which changes the answer to "in which year was ## method machine reading comprehension. the mrc task aims to answer a question based on the context, where the context might be hybrid in complex scenarios, including paragraphs and tables. formally, given a question q, the mrc model is required to reason over the context c and learn a function g(c, q) to predict the labeled answer a. technically speaking, the function g(•) is optimized by fitting the correlation from c and q to a. however, there widely exist spurious correlations @xcite in the complex context. learning from such spurious correlations will ignore the semantic of c and q that causally decide the answers, leading to poor generalization ability. ## experiments in this section, we conduct experiments to answer the following research questions: rq1: how does the proposed htf perform on removing spurious correlations? rq2: how do the regularization terms of htf influence its effectiveness? rq3: how does htf improve the mrc model regarding different spurious correlations? ## related work counterfactual training. stemming from the causal theory @xcite , counterfactual training has become a popular approach recently to avoid learning spurious correlation by doing interventions on the observed data. counterfactual examples have been applied to a wide range of task such as natural language inference @xcite , named entity recognition @xcite , visual question answering @xcite @xcite , story generation @xcite , mrc @xcite ), text classification @xcite , language representation @xcite and information extraction @xcite . researchers also apply the idea of counterfactual into designing training or inference frameworks @xcite @xcite @xcite @xcite @xcite . apart from obtaining counterfactual examples via human-annotation, researcher also study automatically generating counterfactual examples @xcite @xcite @xcite . in tabular mrc task, automatically creating counterfactual examples is infeasible and sufficient human knowledge is still essential. we are inspired by the hypothetical questions proposed in @xcite which we think can be an economic alternative for counterfactual tables, and we are the first to study removing spurious correlations with hypothetical examples. spurious correlation. the problem of spurious correlation has been studied by a wide range of machine learning tasks, such as the unimodal bias in vqa @xcite , the position bias of mrc @xcite , the hypothesis-only bias of nli @xcite , the word alignment of passage and options in qa @xcite , the simplicity bias @xcite , all of which hinder the generalization ability of deep models to out-of-distribution test sets (e.g., @xcite ). solutions have been propose to solve the spurious correlation problems apart from the counterfactual training approaches mentioned above, such as capturing and then mitigating the bias @xcite @xcite , training multiple models @xcite @xcite , invariant learning @xcite , instance mixup @xcite , and using causal inference techniques @xcite . ## conclusion in this work, we investigated the spurious correlations in mrc with tabular context. we proposed to use hypothetical examples for hypothetical training, which teaches the mrc model the effect of the semantic intervention on causing answer changes. by learning such effect, mrc models could effectively remove the spurious correlations and achieve superior generalization performance on a stress test and another tabular mrc dataset. this work leaves many promising directions for future exploration: 1) adopting htf to other language understanding and reasoning tasks that are costly to construct counterfactual examples; 2) expanding htf to model the semantic relationships between multiple hypothetical examples; and 3) applying hypothetical training to various domains apart from the financial domain.