Cheater’s Bowl: Human vs. Computer Search Strategies for Open-Domain QA (2022)

Abstract: For humans and computers, the first step in answering an open-domain question is retrieving a set of relevant documents from a large corpus. However, the strategies that computers use fundamentally differ from those of humans. To better understand these differences, we design a gamified interface for data collection—Cheater’s Bowl—where a human answers complex questions with access to both traditional and modern search tools. We collect a dataset of human search sessions, analyze human search strategies, and compare them to state-of-the-art multi-hop QA models. Humans query logically, apply dynamic search chains, and use world knowledge to boost searching. We demonstrate how human queries can improve the accuracy of existing systems and propose improving the future design of QA models.

## how humans and computers search to compare how humans and computers form queries to answer questions, we first need to have a level playing field and set up our vocabulary. sometimes, we will need to speak abstractly about who is trying to answer the question without distinguishing between the human and the computer. in these cases, we refer to them as an "agent", which can be either the human or the computer. we assume that the agents do not know the answers directly and that they create text-based queries to find the answer (we discuss the alternatives, closed book qa, directly forming dense queries and other computer systems, in section 6). we assume that agents, given an initial question, form a text query q 0 . the i th query q i retrieves a set of documents d i+1 = {d 1 , . . . , d |d i+1 | } from a large corpus of documents d-in our case all the paragraphs in wikipedia pages. the retrieved documents provide additional information, allowing the agent to answer the question or compose a new query q i+1 . the set of documents e i ⊆ d i provide information-evidence-for answering the question with answer a or composing subsequent queries {q j |@xmath0. it is possible that not all retrieved documents are read-e i ̸ = d i -since not all of the retrieved documents are relevant, and an agent might only read a few of them. this process repeats until the agent answers the question. we represent the iterative question-answering process as action path: ## cheater's bowl: gamified data collection for human searches this section discusses the gamified data collection process via an example. ## human vs. computer search strategies this section compares and contrasts how computers and humans search for information. ## existing models and future design although we present queries suggested by stateof-the-art multi-hop qa models to players, players would rather write their own queries (figure 4 ). most players understand why qa models query the way they do (figure 5 ) and agree that queries retrieve helpful results, but players question the utility. this is an intrinsic difference between humans and models: human queries strive for a "direct hit" with two to three search results, as @xcite find that humans only access results on the first page, typically the first few results. in contrast, verbose model queries hope search results contain something helpful-it does not mind reading through a dozen search results. another reason might be that qa models are worse than humans: for qb questions randomly given to players, 56.58% of the questions are correctly answered by players, while only 44.21% are correctly answered by irrr. 3 3 for questions randomly sampled from hotpotqa, human accuracy (71.43%) is slightly lower than irrr's 79.02%. we consider this to be due to the synthetic construction of the hotpotqa dataset: it lends itself to straightforward searches. qb better discriminates (by design) between agents' ability. ## conclusion open-domain and multi-hop qa is an important problem for both humans and computers. to compare how humans and computers search and answer complex questions, our interface collects human question answering data as agents search with traditional and neural search engines alongside question answering models that suggest queries and answers. humans often use shorter queries, apply dynamic search chains, and use world knowledge. future qa models should have the ability to generate novel queries, "discard" irrelevant results, and explicitly check answers. moreover, computer agents for qa should also be able to use diverse retrievers to find evidence to answer questions, learning from the insights found in human data. with an agent trained on our data, we could have the "best of both worlds" to combine the ingenuity and tacit knowledge of humans with an indefatigable agent with access to all the world's information.