Locke’s Holiday: Belief Bias in Machine Reading (2021)

Abstract: I highlight a simple failure mode of state-of-the-art machine reading systems: when contexts do not align with commonly shared beliefs. For example, machine reading systems fail to answer What did Elizabeth want? correctly in the context of ‘My kingdom for a cough drop, cried Queen Elizabeth.’ Biased by co-occurrence statistics in the training data of pretrained language models, systems predict my kingdom , rather than a cough drop . I argue such biases are analogous to human belief biases and present a carefully designed challenge dataset for English machine reading, called Auto-Locke, to quantify such effects. Evaluations of machine reading systems on Auto-Locke show the pervasiveness of belief bias in machine reading.

## introduction reading comprehension models are biased in many ways: they often expect lexical overlap between answer and question @xcite , expect the answers to occur in specific positions @xcite , or expect answers to be named entities @xcite . this paper considers belief bias @xcite in the context of machine reading based on language models. belief bias is a type of cognitive bias, defined in psychology as the tendency to evaluate a statement based on prior beliefs rather than its logical strength @xcite . in figure 1 , the answer ('germany') follows straightforwardly from the context (without inference), but machine reading models nevertheless err, presumably because the prediction ('malaysia') aligns better with associations learned by language models. in another example, models are presented with the following context: 'washington is a number. boston is a city.' in this context, the models evaluated below were unable to answer * each author wishes the others had contributed more. ## machine reading we briefly present the four (common, popular) machine reading models evaluated in §3 and §4: ## data hand-crafted challenges to highlight the failure mode, i created 20 context-question pairs similar to the example in figure 1 . each context had to consist of exactly two clauses, with two binary predicates and at least three distinct arguments. in figure 1 , the first clause expresses a binary isarelation between washington and number; the second clause expresses a binary isa-relation between boston and city. each question had to contain exactly one predicate and one argument, e.g., what is washington?, in effect querying for the missing argument. i list the full set of examples in note that the examples have very short contexts and thereby a small set of potential candidate answers (as models are restricted to return a context span as answer), and they require no or very little reasoning. the example in figure 1 , of course, requires no reasoning at all. neither does any of the first 5 examples. other examples require minimal reasoning, e.g., application of lexical synonymy, anaphora resolution, or ellipsis, such as with the following gapping construction (example 11):