What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2020)

Abstract: Open-domain question answering (QA) involves many knowledge and reasoning challenges, but are successful QA models actually learning such knowledge when trained on benchmark QA tasks? We investigate this via several new diagnostic tasks probing whether multiple-choice QA models know definitions and taxonomic reasoning—two skills widespread in existing benchmarks and fundamental to more complex reasoning. We introduce a methodology for automatically building probe datasets from expert knowledge sources, allowing for systematic control and a comprehensive evaluation. We include ways to carefully control for artifacts that may arise during this process. Our evaluation confirms that transformer-based multiple-choice QA models are already predisposed to recognize certain types of structural linguistic knowledge. However, it also reveals a more nuanced picture: their performance notably degrades even with a slight increase in the number of “hops” in the underlying taxonomic hierarchy, and with more challenging distractor candidates. Further, existing models are far from perfect when assessed at the level of clusters of semantically connected probes, such as all hypernym questions about a single concept.

## introduction automatically answering questions, especially in the open-domain setting where minimal or no contextual knowledge is explicitly provided, requires considerable background knowledge and reasoning abilities. for example, answering the two questions in the top gray box in figure 1 requires identifying a specific isa relation (that 'cooking' is a type of 'learned behavior') as well as recalling a concept definition (that 'global warming' is defined as a 'worldwide increase in temperature'). recent success in qa has been driven largely by new benchmarks @xcite @xcite and advances in model pre-training @xcite . this raises a natural question: do state-of-the-art multiple-choice qa (mcqa) models that excel at standard benchmarks truly possess basic knowledge and reasoning skills expected in these tasks? answering this question is challenging because of limited understanding of heavily pre-trained complex models and the way existing mcqa datasets are constructed. we focus on the second aspect, which has two limitations: large-scale crowdsourcing leaves little systematic control over question semantics or requisite background knowledge @xcite , while questions from real exams tend to mix multiple challenges in a single dataset, often even in a single question @xcite . to address this challenge, we propose systematically constructing model competence probes by exploiting structured information contained in expert knowledge sources such as knowledge graphs and lexical taxonomies. importantly, these probes are diagnostic tasks, designed not to impart new knowledge but to assess what models trained on standard qa benchmarks already know; as such, they serve as proxies for the types of questions that a model might encounter in its original task, but involve a single category of knowledge under various controlled conditions and perturbations. figure 1 illustrates our methodology. we start with a set of standard mcqa benchmark tasks d and a set of models m trained on d. our goal is to assess how competent these models are relative to a particular knowledge or reasoning skill s (e.g., definitions) that is generally deemed important for performing well on d. to this end, we systematically and automatically generate a set of dataset probes p s from information available in expert knowledge sources. each probe is an mcqa rendering of the target information (see examples in figure 1 , yellow box). we then use these probes p s to ask two empirical questions: while our methodology is general, our experiments focus on probing state-of-the-art mcqa models in the domain of grade-school level science, which is considered particularly challenging with respect to background knowledge and inference @xcite @xcite . in addition, existing science benchmarks are known to involve widespread use of definition and taxonomic knowledge (see detailed analysis by @xcite , @xcite ), which is also fundamental to deeper reasoning. accordingly, we use the most widely used lexical ontology wordnet @xcite and publicly available dictionaries as sources of expert knowledge to construct our probes, wordnetqa (section 3.1) and dictionaryqa (section 3.2). 1 these probes measure competence in various settings including hypernymy, hyponymy, and synonymy detection, as well as word sense disambiguation. our exploration is closely related to the recent work of @xcite . however, a key difference is that they study language models (lms), for which there is no clear a priori expectation of specific knowledge or reasoning skills. in contrast, we focus on models heavily trained for benchmark qa tasks, where such tasks are known to require certain types of knowledge and reasoning skills. we probe whether such skills are actually learned by qa models, either during lm pre-training or when training for the qa tasks. recognizing the need for suitable controls in any synthetic probing methodology @xcite , we introduce two controls: (a) the probe must be challenging for any model that lacks contextual embeddings, and (b) strong models must have a low inoculation cost-that is, when fine-tuned on a few probing examples, the model should mostly retain its performance on its original task. 2 this ensures that the probe performance of a model, even when lightly inoculated on probing data, reflects its knowledge as originally trained for the benchmark task, which is precisely what we aim to uncover. constructing a wide range of systematic tests is critical for having definitive empirical evidence of model competence on any given phenomenon. such tests should cover a broad set of concepts and question variations (i.e., systematic adjustments to how the questions are constructed). when assessing isa reasoning, not only is it important to recognize in the question in figure 1 that cooking is a learned behavior, but also that cooking is a general type of behavior or, through a few more inferential steps, a type of human activity. our automatic use of expert knowledge sources allows constructing such high-coverage probes, circumventing pitfalls of solicitation bias and reporting bias. our results confirm that transformer-based qa models foot_2 have a remarkable ability to recognize the types of knowledge captured in our probes-even without additional fine-tuning (i.e., in a zeroshot setting). such models can even outperform strong task-specific non-transformer models trained directly on our probing tasks (e.g., +26% compared to a task-specific lstm). we also show that the same models can be effectively re-finetuned on small samples (even 100 examples) of probe data, and that high performance on the probes tends to correlate with a smaller drop in the model's performance on the original qa task. our comprehensive assessment also reveals important nuances to the positive trend. for example, we find that the best models still perform 2-10% (absolute) below conservative estimates of human performance (section 3.1.3) on these tasks. further, the accuracy of even the best qa model degrades substantially on our hyponym probes (by 8-15%) when going from 1-hop hyponym links to 2-hops. the accuracy on the wordnetqa probe drops by 14-44% under our cluster-level analysis (section 3.1.1), which assesses whether a model knows several facts about each individual concept, rather than only answering correctly isolated questions. this shows that state-of-theart qa models have much room to improve even in some fundamental building blocks (definitions and taxonomic hierarchies) of more complex forms of reasoning. ## related work we follow recent work on constructing challenge datasets for probing neural models, which has primarily focused on the task of natural language inference (nli) @xcite @xcite . most of this work looks at constructing data through adversarial generation methods, which have also been found useful for creating stronger models @xcite . there has also been work on using synthetic data of the type we consider in this paper @xcite @xcite . we closely follow the methodology of @xcite , who use hand-constructed linguistic fragments to probe nli models and study model re-training using a variant of the inoculation by fine-tuning strategy of @xcite . in contrast, we focus on probing open-domain mcqa models (see @xcite for a study on reading comprehension) as well as constructing data from much larger sources of structured knowledge. our main study focuses on probing the bert model and fine-tuning approach of @xcite , and other variants thereof, which are all based on the transformer architecture of @xcite . there have been recent studies into the types of relational knowledge contained in large-scale knowledge models @xcite @xcite , which also probe models using structured knowledge sources. these studies, however, primarily focus on unearthing the knowledge contained in the underlying language models as is without further training, using simple (single token) cloze-style probing tasks and templates. most of these results only provide a lower-bound estimate of model performance, since the probing templates being used potentially deviate from what the model has observed during pre-training. in contrast, we focus on understanding the knowledge contained in language models after they have been trained for a qa end-task using benchmark datasets in which such knowledge is expected to be widespread. further, our evaluation is done before and after these models are fine-tuned on our small samples of target data. this has the advantage of allowing each model to become informed about the format of each probe. we also explore a more complex set of probing templates. the use of lexical resources such as wordnet to construct datasets has a long history, and has recently appeared in work on adversarial attacks @xcite and general task construction (pilehvar and camacho-collados, 2019). in the area of mcqa, there is related work on constructing questions from tuples @xcite , both of which involve standard crowd annotation to elicit question-answer pairs (see also @xcite . in contrast to this work, we focus on generating data in an entirely automatic and silverstandard fashion (i.e., in a way that potentially introduces a little noise), which obviates the need for expensive annotation and gives us the flexibility to construct much larger datasets that control a rich set of semantic aspects of the target questions. following standard practices in mcqa dataset creation (e.g., @xcite , however, we perform crowd-sourcing to obtain conservative (in the sense of @xcite ) estimates of human performance on our main evaluation sets, to compare against model performance. although our probing methodology is amenable to any domain, we focus on probing open-domain qa models in the domain of grade-school level science using a standard suite of benchmark qa datasets (see we also examined mcqa models trained on closely related datasets tailored to commonsense and situational reasoning @xcite @xcite . however, there has been a limited study of the kinds of knowledge needed in this domain, as well as expert knowledge sources for creating corresponding probes. mcqa models trained in this domain exhibit lower performance on our definition and isa probes. ## dataset probes and construction our probing methodology starts by constructing challenge datasets (figure 1 , yellow box) from a target set of knowledge resources. each probing dataset consists of multiple-choice questions that include a question q and a set of answer choices or candidates {a 1 , ...a n }. this section describes in detail the 5 datasets we build (grouped into wordnetqa and dictionaryqa), drawn from two publicly available resources: wordnet @xcite and the gnu collaborative international dictionary of english (gcide). 4 for convenience, we will describe each source of expert knowledge as a directed, edge-labeled graph g. the nodes of this graph are @xmath0 set wordnet (wn) gcide r {isa↑,isa↓, {def, ex, lemma} def, ex, lemma} c {wn synsets} {entry ids} d {synset glosses} {unique defs} s {synset sentences} {entry examples} w {synset lemmas} {all words} atomic triple types definition concept senses and definitions t d ⊆ {def} × c × d concepts with example sentences te ⊆ {ext} × c × s concepts with words t l ⊆ {lemma} × c × w isa relations (wn only) t i ⊆ {isa ↑ ,isa ↓ } × c × c w ∪ s ∪ d, where c is a set of atomic concepts, w a set of words, s a set of sentences, and d a set of definitions (see when defining our probe question templates, it will be useful to view g as a set of (relation, source, target) triples t ⊆ r × c × v. because of their origin in an expert knowledge source, such triples preserve semantic consistency. for instance, when the relation in a triple is def, the corresponding edge maps a concept in c to a definition in d. we rely on two heuristic functions, defined below for each individual probe: gen q (τ ), which generates gold question-answer pairs (q, a) from a set of triples τ ⊆ t and question templates q, and distr(τ ′ ), which generates distractor answers choices {a ′ 1 , ...a ′ n -1 } based on another set of triples τ ′ (where usually τ ⊂ τ ′ ). for brevity, we will use gen(τ ) to denote gen q (τ ). in generating our dataset probes, our general strategy is to build automatic silver-standard training and developments sets, in the latter case at a large scale to facilitate detailed and controlled analysis of model performance. as discussed below, we also provide estimates of human performance on our test sets, and in some cases introduce smaller gold-standard test sets to allow for a direct comparison with model performance. ## probing methodology and modeling given the probes above, we now can start to answer the empirical questions posed at the beginning. our main focus is on looking at transformer-based mcqa models trained on science benchmarks in ## results and findings we begin with an assessment to ensure that our probes are sufficiently difficult to provide meaningful insights into strong models (section 5.1), then assess the strength of pre-trained qa models (section 5.2) and whether they can be effectively inoculated (section 5.3), and finally present a cluster-based consistency analysis (section 5.4). ## discussion we presented a new methodology for automatically building challenge datasets from knowledge graphs and taxonomies. we introduced several new silver-standard datasets for systematically probing state-of-the-art open-domain qa models. although our focus was on probing definitions and isa reasoning, the methodology is amendable to any target knowledge resource or qa domain. we see synthetic datasets and our general methodology as an inexpensive supplement to recent large-scale investment in naturalistic qa dataset construction @xcite to help better understand today's models. we found transformer-based qa models to have a remarkable ability to reason with complex forms of relational knowledge, both with and without exposure to our new tasks. in the latter case (zero-shot), a newer roberta qa model trained only on benchmark data outperforms several task-specific lstm-based models trained directly on our probes. when inoculated using small samples (e.g., 100 examples) of probing data, roberta masters many aspects of our probes with virtually no performance loss on its original qa task-which we use as a control on the probing quality. because these models seem to already contain considerable amounts of relational knowledge, our simple inoculation strategy, which nudges models to bring out this knowledge explicitly while retaining performance on their original task (hence allowing a fairer probe of its knowledge by giving the model the opportunity to learn the probe format), could serve as a simpler alternative to designing new model architectures explicitly encoding such knowledge @xcite . regarding our focus on preserving a model's performance on its original task, one might expect that re-training on relevant knowledge should improve performance. following other work in this area @xcite , we found that maintaining performance after additional fine-tuning on specialized datasets is already a tall order given that models are susceptible to over-specialization; indeed, similar issues have been noticed in recent work on largescale transfer learning @xcite . we believe that using inoculation for the sole purpose of improving model performance, which is beyond the scope of this paper, would likely require a more sophisticated inoculation protocol. devising more complex loss functions extending our inoculation strategy to help balance old and new information could help in this endeavor. the main appeal of automatically generated probes is the ability to systematically manipulate probe complexity, which in turn enables more controlled experimentation as well as new forms of evaluation. it allowed us to study in detail the effect of different types of distractors and the complexity of required reasoning. this study showed that even the best qa models, despite additional fine-tuning, struggle with harder categories of distractors and with multi-hop inferences. for some probes, our cluster-based analysis revealed that errors are widespread across concept clusters, suggesting that models are not always consistent and robust. these results, taken together with our findings about the vulnerability of synthetic datasets to systematic biases and comparison with human scores, suggest that there is much room for improvement and that the positive results should be taken with a grain of salt. developing better ways to evaluate semantic clusters and model robustness would be a step in this direction. we emphasize that using synthetic versus naturalistic qa data comes with important tradeoffs. although we are able to generate large amounts of systematically controlled data at virtually no cost or need for manual annotation, it is much harder to validate the quality of such data at such a scale and such varying levels of complexity. conversely, with benchmark qa datasets, it is much harder to perform the type of careful manipulations and cluster-based analyses we report here. while we assume that the expert knowledge we use by virtue of being hand-curated by human experts, is generally correct by design, we know that such resources are fallible and errorprone. we propose measuring human performance via small samples of probing data, and leave more scalable methods of removing potential noise and adding human annotation to future work. one of the overarching goals of our approach to model probing is to uncover whether black box models are able to reason in a consistent and correct manner. our assumption, similar to @xcite , is that the ability of a model to mimic the input-output behavior of data generated using expert knowledge gives some evidence of correctness in virtue of such data being correct by construction (see discussion by @xcite ). we emphasize, however, that there are limits to how much we can learn through this type of behavioral testing, given that models are susceptible to exploiting systematic biases in synthetic data and the general difficulty of disentangling a model's knowledge acquired during pre-training versus fine-tuning @xcite . we therefore see efforts to combine behavioral testing with various other analysis methods @xcite that aim to uncover correlations and causal patterns between internal model representations and discrete structures @xcite @xcite as a promising direction for future work. this, in combination with extending our probing strategy to other forms of expert knowledge, could prove to be an effective way to engage others working on linguistics and other areas of ai in state-of-the-art nlp research.