Evaluation Paradigms in Question Answering (2021)

Abstract: Question answering (QA) primarily descends from two branches of research: (1) Alan Turing’s investigation of machine intelligence at Manchester University and (2) Cyril Cleverdon’s comparison of library card catalog indices at Cranfield University. This position paper names and distinguishes these paradigms. Despite substantial overlap, subtle but significant distinctions exert an outsize influence on research. While one evaluation paradigm values creating more intelligent QA systems, the other paradigm values building QA systems that appeal to users. By better understanding the epistemic heritage of QA, researchers, academia, and industry can more effectively accelerate QA research.

## introduction this position paper seeks to answer the question why do we do question answering and understand the consequences of different answers to this question. our primary contribution is to outline two distinct and common reasons that motivate researchers to pursue question answering (qa)-the cranfield and manchester paradigms. the cranfield paradigm is not new (section 2): it has a long and storied history in information retrieval @xcite . here, we describe why a large share of qa is implicitly motivated by serving the needs of users, which is exactly the cranfield paradigm (although most do not say so explicitly). section 3 christens another paradigm-the manchester paradigm-at home in the more eclectic corners of qa: to test and inculcate intelligence. these paradigms have much in common (section 4), which helps explains why this distinction is not immediately apparent. however, the differences (section 5) are ignored at your own peril. section 6 articulates how the community can better heed the distinction and how the paradigms can inform each other. in information retrieval, a "system locates information that is relevant to a user's query" @xcite . the most natural ir evaluation is to ask users whether documents satisfy their information need. however, much like annotation in nlp, this is expensive and time-consuming, so @xcite proposes an alternative. rather than have users interact with every potential system, build re-usable test collections and evaluate all systems by re-using the same collection. although "obvious" to twenty-first century readers, the use of offline test collections for evaluation was controversial @xcite , and the approach is still debated @xcite . rather than putting users in front of every ir system, in the cranfield paradigm, researchers perform experiments on test collections to compare the relative effectiveness of different retrieval approaches @xcite cranfield paradigm datasets approximate users' searches, and the better your algorithm satisfies those queries, the better @xcite the algorithm. as ir systems conquered retrieving documents for short queries, researchers turned to finding short answers instead of whole documents @xcite , which naturally lends itself to answering questions and "move[s] retrieval systems closer to information retrieval as opposed to document retrieval" @xcite . under the cranfield paradigm, a good qa system should answer the questions users ask. what more could you want? it is then unsurprising that google and microsoft adopted this setting for natural questions @xcite and ms marco @xcite . they found questions people asked online and answered them. a good qa system should automate that process @xcite . ## probing and pushing answerers: the manchester paradigm in the manchester paradigm, we create tasks and datasets whose questions push answerers to better understand the world and create evaluations that probe for human-like capabilities. since we identify and name the paradigm, we give three justifications that highlight three distinct reasons people ask questions beyond information seeking: to teach, to compare, and to probe. ## what cranfield and manchester share although these paradigms have different core goals, research advancing the goals of one often advances the goals of the other. while there are differences between qa datasets across paradigms @xcite @xcite , these differences are overshadowed within a paradigm by dataset-specific quirks. thus, a paradigmagnostic blueprint for qa @xcite is to combine sparse @xcite or dense retrieval @xcite followed by span selection @xcite or generation @xcite . as a consequence, researchers indifferent to which questions are answered can improve representations and algorithms for both paradigms (although as interactions become richer, this may not be the case, as we discuss at the end of section 6). the paradigms' evaluations also overlap; they benefit from expert annotators @xcite , crowd annotators, and alternative evaluations like behavioural testing @xcite . similarly, both paradigms value robustness @xcite . additionally, answering infrequently asked questions is important for search engines @xcite , and building models that learn more from less qualifies as intelligent behaviour (linzen, 2020). creating systems robust to spelling mistakes @xcite is a worthy goal. from the cranfield perspective, systems hobbled by spelling mistakes lead to a poor user experience. on the other side, humans are impressively robust to poor spelling @xcite , so from the manchester perspective this form of robustness is also valuable. but this has its limits; in the next section, we argue why adversarial examples are more consistent with the manchester paradigm. users are the customers many future business dissertations will survey ibm watson's circuitous route from trec system @xcite to jeopardy! spectacle to embattled spin-off @xcite despite "ibm @xcite bragged to the media that watson's questionanswering skills are good for more than annoying alex trebek." @xcite . one challenge may have been transitioning between paradigms. one aspect that made the transition more difficult was that the tour de force victory on jeopardy! was firmly on the side of the manchester paradigm, but to be a successful commercial application, it needed to make the shift to the cranfield paradigm. similarly, squad was written by people (mechanical turkers) who knew the answers. . . just like most of the questions in the manchester paradigm. however, it did not follow the same principles of the manchester paradigm, which led to the "shortcuts" that other investigators have discovered in the years since @xcite . for example, priming made exploitable clues more frequent and mechanical turkers write each question as quickly as possible. @xcite anticipates this behaviour, specifically avoiding "cheap tricks" in their manchester-paradigm winograd challenge. in other manchester paradigm questions, trivia question writers frequently take pride in wellcrafted questions @xcite . comparisons one of the primary inspirations for the manchester paradigm is competitions (e.g., university challenge). because these competitions are meant to determine who the smartest answerer is, they are remarkably efficient. the world accepted the judgement that watson was smarter than ken jennings and brad rutter after 122 answers. why not? these competitions are designed to discriminate between player abilities. in contrast, the dev and test sets of cranfield-inspired datasets have thousands of questions, and even that may not be enough @xcite . ## call to action our central plea is that researchers in qa and nlp more broadly should have a clear answer to the question: "why are you working on this?". this is of particular importance as qa datasets proliferate @xcite , and nlp practitioners "lost in dozens of recent datasets" want to know what datasets measure @xcite . @xcite offer a trenchant enumeration of qa uses, foot_2 we think the onus of definition should fall on dataset creators, not on post-hoc analyses. other than more explicitly naming two of the uses of qa after english university towns, our goal is to encourage researchers to recognize the tensions between these two uses and the opportunities created from recognizing the distinction. make what you value explicit each of these paradigms value different skills and embed these values in datasets and tasks. to make machine learning useful to society and adopt valuesensitive design @xcite , developers of datasets should make their goals clear from the outset @xcite . in the cranfield paradigm, aligning these evaluations with user satisfaction is essential @xcite . industry is naturally financially motivated towards this goal, and they have the user data @xcite -only a fraction of which is published to protect privacy @xcite . still, strategic and thoughtful partnerships like the cranfield-inspired trec workshops are valuable; without trec, it is estimated that "us internet users would have spent up to 3.15 billion additional hours using web search engines between 1999 and 2009" @xcite . one of the goals of the manchester paradigm should be to identify the linguistic phenomena or ethnic and linguistic groups @xcite that are not well-served by cranfieldfocused data. thus, before you begin your question answering research, make it clear what your goal is: are you trying to build agi foot_3 or to serve users? that answer will then inform your evaluation methology. academia's special role it is no coincidence that our paradigms are named after the homes of universities, and universities are where the manchester paradigm will thrive. thus, there is a strategic opportunity for academia and funding agencies to support manchester-aligned work abjured by deep-pocketed industry. lovingly crafted questions by trivia experts @xcite and adversarial questions @xcite @xcite are unlikely to change the way a smart assistant answers a question, but they might expose blind spots of qa systems or improve evaluation. moreover, asking questions in public is not just entertaining; it can generate data @xcite and help the public better understand the possibilities and limitations of ai @xcite . thus, those in the manchester paradigm can game show-ify evaluations to make question answering more fun and illuminating. build for the future we do not advocate for firewalling these interests: they are ideally synergistic. cranfield-inspired tasks can identify the most helpful capabilities that manchester-inspired tasks can work towards. however, evaluating systems on users' current information needs may leave much on the table. users' habits and low expectations encourage users to avoid difficult questions @xcite : e.g., avoiding complex syntax or hard to recognize named entities @xcite with voice recognition. begin a dialog with users regardless of which paradigm you favor, qa is at its heart an interaction with users. in the cranfield paradigm, the user knows less than the system. in the manchester paradigm, the user knows more and takes the role of a teacher or an evaluator. in both cases, shneiderman (2021) argues that responsible ai should enable an interactive, responsive conversation between the system and the user. in the cranfield paradigm, this is an opportunity to correct false presuppositions: "when did raphael paint the mona lisa" could flag that da vinci painted it in 1503 and to explain multiple interpretations of a question @xcite . in the manchester paradigm, this can use dialog to train systems @xcite , guide the system to semantically equivalent answers @xcite , or to learn from how humans answer the same questions @xcite . for example, if a computer answers bush to the question "who appointed scalia to the supreme court", a manchester inquisitor would rightly follow up with "can you be more specific", to which the system would hopefully respond george w. bush. ## conclusion we identify two core motivations for qa research over the past twenty years. we link one to the usercentered goals of the cranfield paradigm and propose the manchester paradigm to describe research working towards building human-like, intelligent qa systems. in at least the short-term, this distinction is important as it illuminates the goals of industry and academic stakeholders; ultimately, this makes it easier to ensure that both research agendas are valued. in the long term, we suspect that the best qa agents will benefit from the insights of user-oriented tasks and the longer-range efforts towards natural language understanding (bender and koller, 2020; linzen, 2020).