SD - QA : Spoken Dialectal Question Answering for the Real World (2021)

Abstract: Question answering (QA) systems are now available through numerous commercial applications for a wide variety of domains, serving millions of users that interact with them via speech interfaces. However, current benchmarks in QA research do not account for the errors that speech recognition models might introduce, nor do they consider the language variations (dialects) of the users. To address this gap, we augment an existing QA dataset to construct a multi-dialect, spoken QA benchmark on five languages (Arabic, Bengali, English, Kiswahili, Korean) with more than 68k audio prompts in 24 dialects from 255 speakers. We provide baseline results showcasing the real-world performance of QA systems and analyze the effect of language variety and other sensitive speaker attributes on downstream performance. Last, we study the fairness of the ASR and QA models with respect to the underlying user populations.

## introduction the development of question answering @xcite systems that can answer human prompts with, or in some cases without, context is one of the great success stories of modern natural language processing (nlp) and a rapidly expanding research area. usage of such systems has reached a global scale in that millions of users can conveniently query voice assistance like google assistant, amazon alexa, or apple siri through their smartphones or smart home devices. qa systems have also made large strides in technology-adjacent industries like healthcare, privacy, and e-commerce platforms. wide-spread adoption of these systems requires they perform consistently in real-world conditions, an area where @xcite note there  asr transcription: teolojia ya dogma ni nini?  qa minimal answer: neno linalotumika hasa kumaanisha fundisho la imani lisiloweza kukanushwa na wafuasi wa dini fulani. figure 1 : illustration of the envisioned scenario for a user-facing qa system that sd-qa aims to evaluate (example from kiswahili). in this work, we address the need for realistic evaluations of qa systems by creating a multilingual and multi-dialect spoken qa evaluation benchmark. our focus is the utility of qa systems on users with varying demographic traits like age, gender, and dialect spoken. our contributions are as follows: 1. we augment the tydi-qa dataset @xcite with spoken utterances matching the questions. in particular, we collect utterances in four languages (arabic, bengali, english, kiswahili) and from multiple varieties 2 (seven for arabic, eleven for english, and two for each of bengali, kiswahili, and korean). 2. we perform contrastive evaluation for a baseline pipeline approach that first transcribes the utterances with an asr system and then provides the transcription to the qa system. we compare general and localized asr systems, finding wide divergences between the downstream qa performance for different language varieties. the sd-qa dataset builds on the tydi-qa dataset @xcite , using questions, contexts, and answers from five typologically diverse languages. our focus is to augment the dataset along two additional dimensions. the first is a speech component, to match the real-world scenario of users querying virtual assistants for information. second, we add a dimension on dialectal and geographical language variation. ## tasks and evaluation we perform three tasks over our dataset, defined below. the passage selection and minimal answer selection tasks are directly modeled after the primary task in tydi-qa: 1. automatic speech recognition (asr) task: a standard task defined over the utterances of the different language varieties. given the audio file of the utterance, the model has to produce an accurate transcription. since our questions are parallel across all language varieties, sd-qa can be used for contrastive evaluation of the robustness of asr systems across different varieties. 2. passage selection task: given the question and a number of candidate passages, this task i ii iii systems accuracy variety a variety b figure 2: schematic visualization of three systems tested on two language varieties. system (ii) is preferable to system (i) since it is more equitable. system (iii) is the ideal scenario. ## baseline results and discussion baseline models we benchmark the speech recognition systems using the asr models through the google speech-to-text api. 9 for all language varieties, we follow a pipeline approach, where we first transcribe the audio utterance and use the output as the input (question) to the qa model. we leave end-to-end multimodal approaches that operate directly on the audio for future work. our qa model follows the recipe of @xcite , training a single model for all languages using multilingual bert @xcite as a feature extractor. the model is trained for a maximum of 10 epochs, selecting the best checkpoint based on development set f 1 -score. ## related work due to space limitations, we provide a detailed account of related work on qa benchmark datasets and on multilingual qa data and approaches in appendix h. we discuss here, though, the most relevant works on spoken qa. speech qa a number of recent studies are done to bridge the gap between text based qa system and speech data. in spoken squad @xcite , the authors propose a new task where the question is in textual form but the related reading comprehension is given in speech form. so transcription is performed on the speech data and the output can be in either text form or audio time span. even using state-of-theart speech transcription model, the authors show severe deterioration in performance. the authors further propose subword unit sequence embedding based mitigation strategies. this work was further extended to the odsqa dataset @xcite , where the question is also given in speech the most relevant work on the intersection of speech and qa is the recent noiseqa @xcite study, which explores the effect of noise in qa systems introduced by 3 main input interfaces: keyboard input, machine translation and speech input. our task of assessing speech input error is conceptually similar to their speech input assessment. their experiments show that the absence of punctuation in asr outputs results in performance degradation by 5.1%. in addition, voice variation, accent and speaker's acoustic conditions as well as choice of asr unit also play an important role. this study also shows that transcription of naturally spoken question results in errors like question type shift, ungrammatical or meaningless questions, corrupted named entities and dropped delimiters. @xcite make a number of recommendations including assessing the source of error, context-driven evaluation and community priorities while designing robust qa systems. we believe that our work is a necessary expansion and complement of such studies. we go beyond @xcite by providing real audio data in more than one language instead of synthetic text-to-speech data.furthermore, we go beyond english by providing data in four more languages and several varieties/locales. ## conclusion we present sd-qa, a new benchmark for the evaluation of qa systems in real-world settings, mimicking the scenario of a user querying a qa system through a speech interface. sd-qa is the largest spoken multilingual and multi-dialect qa dataset to date, with coverage of five languages and twenty-four dialects. we provide baseline performance and fairness results on a pipeline that uses publicly available asr models to transcribe spoken queries before passing them to a multilingual qa system. we showcase the qa systems' lack of robustness to noise in the question. we also discuss the fairness of both speech recognition and qa models with regards to underlying user characteristics, and show that a user's dialect can significantly impact the utility they may receive from the system. future areas of improvement for this work include expanding sd-qa to cover more languages and dialects, and additional analysis on attributes about the data and users. ideally, we would like to discern which parameters most influence the performance of downstream language systems. we also plan to investigate the prospect of an end-to-end spoken qa system in an attempt to bridge the gap between the speech modality of the query and the textual modality of the currently available knowledge bases. ethical considerations all annotators involved in the collection of sd-qa signed informed consent forms about the nature of this work and were compensated fairly ($15/hour) for their work. the data collection process and its limitations are discussed in section §2.2. the released dataset has been anonymized, while all demographic/identity characteristics are self-reported. the final release of the dataset includes an accompanying data statement @xcite .