Using counterfactual contrast to improve compositional generalization for multi-step quantitative reasoning (2023)

Abstract: In quantitative question answering, compositional generalization is one of the main challenges of state of the art models, especially when longer sequences of reasoning steps are required. In this paper we propose CounterComp, a method that uses counterfactual scenarios to generate samples with compositional contrast. Instead of a data augmentation approach, CounterComp is based on metric learning, which allows for direct sampling from the training set and circumvents the need for additional human labels. Our proposed auxiliary metric learning loss improves the performance of three state of the art models on four recently released datasets. We also show how the approach can improve OOD performance on unseen domains, as well as unseen compositions. Lastly, we demonstrate how the method can lead to better compositional attention patterns during training.

## introduction enterprise documents such as reports, forms, and analytical articles often include quantitative data in tabular form. the data in these tables can be self-contained, but more commonly the surrounding text provides more context that is necessary to understand the content. answering questions over these hybrid tabular/text contexts requires reasoning that combines verbal and quantitative semantics. question answering over quantitative tabular/text data has gained recent traction with the release of datasets such as finqa @xcite , tat-qa @xcite , and hitab @xcite . the table also shows that as the number of steps grows, generating the wrong operator becomes a more dominant mistake than selecting the wrong operand. not only is this error more dominant, but it can also have a more destructive impact on the chain of reasoning, as it can derail the model's hidden representations from that point onward. as an example, our analysis of the finqanet model @xcite output showed that if the model generates an incorrect operator, it is about 30% more likely to commit other errors in the following steps compared to when the model generates an incorrect operand. in this paper, we propose countercomp, an approach that can enhance compositional learning in multi-step quantitative qa. we take inspiration from the symbolic composition of arithmetic operations, and their correspondence to natural language phrases. building on the work on attention alignments from previous studies, we propose an auxiliary metric learning loss that is focused on specific components of the input and output. our sampling strategy is based on counterfactual scenarios. this means that the model learns proper representations for each component based on what-if scenarios. to the best of our knowledge, this is the first study that successfully applies component-wise counterfactual sampling as a metric learning strategy. we show how, when state of the art models are augmented with our auxiliary metric learning loss, they exhibit better performance in cases where multistep reasoning is required. countercomp outperforms current baselines on four recently released datasets, and show stronger performance on ood samples. ## related work the typical architecture of a quantitative qa model is composed of a retriever and a generator @xcite . the retriever identifies the particular context where the answer might be found. ## problem formulation let us consider the example provided by we flag all spans in the question that do not overlap with the facts, i.e. underlined blue segments. those spans serve as candidate intervention spans. in the example from next, we seek a positive and a negative example within the training set. a positive example is a sample in which, despite possible changes in the question, the operators in the output remain consistent with the operators in the anchor. a negative example is a sample in which exactly one output operator is altered, deleted, or added. the given positive and negative example can now be plugged into equation 2. instead of a fixed margin, we use the edit distances mentioned before to dynamically adjust the margin. let nld (i) pos and nld (i) neg be the normalized, token-level levenshtein edit distance between the anchor and the positive example, and the negative example, respectively. we set the margin to: this encourages a larger margin for cases where the anchor is equally similar to the positive and the negative examples, and the model might have a harder time picking up on the nuances of each component. ## results and analysis as ## conclusion in this paper, we presented countercomp, a method that leverages counterfactual contrast to enable metric learning for quantitative qa. we show how using the auxiliary countercomp loss can improve compositional generalization in multistep reasoning tasks, especially as the number of steps grows. due to runtime challenges, we proposed a hybrid offline/online sampling strategy that uses predefined indices for easier lookup operations. this allows us to capture samples that have a contrast of one operator with the anchor. in future studies, we hope to capture contrastive samples with longer perturbation chains. we also hope to examine the effectiveness of counterfactual compositional contrast in other domains such as semantic parsing and question answering over multimodal input. lastly, we hope to extend the use of counter-comp to enhance the performance of the retriever, using the heuristics introduced in section 4 (i.e. by focusing on components in the question that overlap with the facts). this can result in a quantitative qa pipeline that is powered by compositional contrast in an end-to-end fashion.