Challenges in Generalization in Open Domain Question Answering (2022)

Abstract: Recent work on Open Domain Question Answering has shown that there is a large discrepancy in model performance between novel test questions and those that largely overlap with training questions. However, it is unclear which aspects of novel questions make them challenging. Drawing upon studies on systematic generalization, we introduce and annotate questions according to three categories that measure different levels and kinds of generalization: training set overlap, compositional generalization (comp-gen), and novel-entity generalization (novel-entity). When evaluating six popular parametric and non-parametric models, we find that for the established Natural Questions and TriviaQA datasets, even the strongest model performance for comp-gen/novel-entity is 13.1/5.4% and 9.6/1.5% lower compared to that for the full test set – indicating the challenge posed by these types of questions. Furthermore, we show that whilst non-parametric models can handle questions containing novel entities relatively well, they struggle with those requiring compositional generalization. Lastly, we find that key question difficulty factors are: cascading errors from the retrieval component, frequency of question pattern, and frequency of the entity.

## introduction over the last few years we have seen model innovations improving on standard natural language processing (nlp) benchmarks across the board @xcite @xcite . however, it is still clear that we are yet to obtain generalizable language understanding, as recent work has found that adversarial @xcite @xcite and out-of-distribution samples @xcite @xcite remain challenging for existing models across numerous tasks. ## dataset construction in this section, we describe how we process and annotate odqa datasets to enable us to investigate generalization. ## conclusion we study odqa model generalization and categorize unseen questions into three subsets: overlap, comp-gen, novel-entity. treating questions as being compositional, we decompose them into atomic elements based on their semantics. we believe that this decomposition strategy can help future work related to question structure and unification. we evaluated several recent odqa models on these three subsets for three popular datasets. our experimental findings both pinpoint the specific problems when handling different categories of novel questions and shed light on how to compositionally approach the factoid questions in odqa task.