N oise QA : Challenge Set Evaluation for User-Centric Question Answering (2021)

Abstract: When Question-Answering (QA) systems are deployed in the real world, users query them through a variety of interfaces, such as speaking to voice assistants, typing questions into a search engine, or even translating questions to languages supported by the QA system. While there has been significant community attention devoted to identifying correct answers in passages assuming a perfectly formed question, we show that components in the pipeline that precede an answering engine can introduce varied and considerable sources of error, and performance can degrade substantially based on these upstream noise sources even for powerful pre-trained QA models. We conclude that there is substantial room for progress before QA systems can be effectively deployed, highlight the need for QA evaluation to expand to consider real-world use, and hope that our findings will spur greater community interest in the issues that arise when our systems actually need to be of utility to humans.

## introduction everyday users now benefit from powerful qa technologies in a range of consumer-facing applications including health @xcite @xcite @xcite , privacy @xcite @xcite , personal finance @xcite , search @xcite @xcite and dialog agents @xcite . voice assistants such as amazon alexa foot_1 or google home foot_2 have brought natural language technologies to several million homes globally @xcite . yet, even with millions of users now interacting with these technologies on a daily basis, there has been surprisingly little research attention devoted to studying the issues that arise when people use qa systems. traditional qa evaluations do not reflect the needs of many users who can benefit from qa technologies. for example, users with a range of visual and motor impairments now rely extensively on voice interfaces @xcite for efficient text entry. 4 another need is cross-lingual information access, e.g. in scenarios where a speaker of one of the ∼7000 non-english living languages in the world @xcite ) may want to take advantage of an english qa system. foot_4 qa evaluation has to keep up with the different ways in which users may use these systems in practice, and the different users who interact with these systems. keeping these needs in mind, we construct evaluations considering the interfaces through which users interact with qa systems. 6 we analyze errors introduced by three interface types that could be connected to a qa engine: speech recognizers converting spoken queries to text, keyboards used to type queries into the system, and translation systems processing queries in other languages. our contributions are as follows: original question interface synthetic construction natural construction what has a lama determined to do? what has a llama determined to do what has a llama determined to do what has a lama determined to do? wjat has a lsma determined yo do? what has a lama determied to do? what has a lama determined to do? what has a lama decided to do? what is a llama determined to do? as manually construct natural noise challenge sets, by processing squad questions through the specified interfaces. ## motivation modern qa systems often rely on large databases of digital text such as wikipedia as their source of knowledge; such corpora typically contain wellformed text in a high-resource language like english. however, the user's input could come in many different forms: it could be spoken, or written but in another, possibly lower-resource language. to convert these inputs into the format that the system can process, another machine learning system such as a speech recognizer or a machine translation engine is required, and these intermediate systems will inevitably propagate their decoding errors into the qa engine. however, interface errors are not necessarily artifacts of machine learning models: even when the question comes in the desired form (e.g. english text), it has to be communicated to the qa system through a mechanical interface such as a keyboard, and the process of typing can introduce errors such as character substitutions. to be useful in real-world settings, a qa system has to be able to correctly process the input question regardless of the input interface. we simulate the use cases for three interface categories (asr, mt, and keyboard) with different level of human involvement, from fully automatic pipelines to leveraging existing human-generated resources to manual annotation, and evaluate whether the modern qa systems are capable of going from controlled well-formed inputs to real-world scenarios. ## challenge set construction we define a suite of three types of noise perturbations, each imitating noise specific to a category of interfaces, and apply them to the data to create the challenge sets. we choose to add the noise to the questions but not to the context paragraphs, to replicate a realistic scenario of the noise being introduced to the question by the interface through which the user interacts with the qa engine. for each type of noise, we both build a synthetic generator that can introduce noise on a large scale, as well as manually create 'natural' noise challenge sets to imitate real-world noise. our challenge sets are based on squad 1.1 (rajpurkar et al., 2016), foot_6 a large-scale machine comprehension dataset based on wikipedia articles where the answer to each question is a span in a provided context. we choose squad both for its popularity as a benchmark @xcite @xcite and to avoid additional confounds such as unanswerable questions @xcite . 8 we use the standard ∼90k/10k train/development split and construct the challenge sets from the xquad data @xcite , a subset of 1,190 squad development set questions accompanied by professional translations into ten languages. 9 below we discuss each challenge set in more detail. ## experiments we select four qa models that demonstrated strong performance on squad 1.1 foot_13 to be tested under interface distortions: bidaf @xcite , which represents contexts at different levels of granularity using bidirectional attention flow mechanism; its extension bidaf-elmo @xcite augmented with contextualized embeddings; bert @xcite , a bidirectional transformer-based language model @xcite ; and roberta @xcite , a more robustly pre-trained version of bert. ## conclusion in this work, we advocate for qa evaluations that reflect challenges associated with real-world use. in particular, we focus on questions that are written in another language, spoken, or typed, and the noise introduced into them by the corresponding interface (machine translation, speech recognition, or keyboard). we analyze the effect of synthetic and natural noise in each interface and find that these errors can be diverse, nuanced, and challenging for traditional qa systems. although we present an initial exploration of mitigation strategies, our primary contribution lies not in the specific challenge sets we construct or in developing new algorithms, but rather in identifying and describing one class of problems that practical qa systems must consider and providing a framework to measure them. we hope insights derived from our study stimulate research in making qa systems ready to face realworld users. we emphasize three considerations: sources of error: this work studies errors introduced at the interface stage of qa pipelines. these errors are nearly ubiquitous, as users always interact with qa systems through some kind of interface. thus, it is important for qa system designers to be mindful of distortions those might introduce. our analysis can be extended to study the impact of interface-specific factors: for example, how errors vary by keyboard layout (e.g. qwerty vs. dvorak or language-specific layouts like azerty) or preferred way of typing (e.g. using physical keyboards vs. swipe typing). another fruitful area of study could lie in examining the accumulated impact of errors resulting from interface combinations (e.g. machine translation of asr-transcribed queries) and the effects of such interface noise in languages other than english. however, interface distortion represents only one source of error that occurs in practical deployment, and future research would study further sources of variation such as how users may adapt their questions according to the interface used. context-driven evaluation: this work focuses on practical evaluation of qa systems that takes into account the challenges associated with their real-world deployment. we hope to encourage development of future user-centered or participatory design approaches to building qa datasets and evaluations, where practitioners work with potential users to understand user requirements and the contexts in which systems are used in practice. community priorities for qa systems: while leaderboards on established benchmarks have facilitated rapid progress @xcite and bolstered development of a variety of semantic models @xcite @xcite , we call for practitioners to consider the orthogonal direction of system utility in their model design. we believe these subareas to be complementary, and community attention towards both will help produce nlp systems that are both accurate and usable.