T y D i QA : A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020)

Abstract: Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TyDi QA—a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology—the set of linguistic features each language expresses—such that we expect models performing well on this set to generalize across a large number of the world’s languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phenomena that would not be found in English-only corpora. To provide a realistic information-seeking task and avoid priming effects, questions are written by people who want to know the answer, but don’t know the answer yet, and the data is collected directly in each language without the use of translation.

## introduction when faced with a genuine information need, everyday users now benefit from the help of automatic question answering (qa) systems on a daily basis with high-quality systems integrated into search engines and digital assistants. their questions are information-seeking-they want to know the answer, but don't know the answer yet. recognizing the need to align research with the impact it will have on real users, the community has responded with datasets of informationseeking questions such as wikiqa @xcite , ms marco @xcite , quac pronounced tie dye q. a.-like the colorful t-shirt. ␆ project design ␅ modeling ␄ linguistic analysis ␃ data quality. @xcite , and the natural questions (nq) @xcite . however, many people who might benefit from qa systems do not speak english. the languages of the world exhibit an astonishing breadth of linguistic phenomena used to express meaning; the world atlas of language structures @xcite categorizes over 2,600 languages foot_0 by 192 typological features including phenomena such as word order, reduplication, grammatical meanings encoded in morphosyntax, case markings, plurality systems, question marking, relativization, and many more. if our goal is to build models that can accurately represent all human languages, we must evaluate these models on data that exemplifies this variety. in addition to these typological distinctions, modeling challenges arise due to differences in the availability of monolingual data, the availability of (expensive) parallel translation data, how standardized the writing system is variable spacing conventions (e.g., thai), and more. with these needs in mind, we present the first public largescale multilingual corpus of information-seeking question-answer pairs-using a simple-yet-novel data collection procedure that is model-free and translation-free. our goals in doing so are: we describe the typological features of tydi qa's languages and provide glossed examples of some relevant phenomena drawn from the data to provide researchers with a sense of the challenges present in non-english text that their models will need to handle (section 5). we also provide an open-source baseline model foot_2 and a public leaderboard foot_3 with a hidden test set to track community progress. we hope that enabling such intrinsic and extrinsic analyses on a challenging task will spark progress in multilingual modeling. the underlying data of a research study can have a strong influence on the conclusions that will be drawn: is qa solved? do our models accurately represent a large variety of languages? attempting to answer these questions while experimenting on artificially easy datasets may result in overly optimistic conclusions that lead the research community to abandon potentially fruitful lines of work. we argue that tydi qa will enable the community to reliably draw conclusions that are aligned with people's information-seeking needs while exercising systems' ability to handle a wide variety of language phenomena. figure 1 shows an example question-answer pair. this formulation reflects that informationseeking users do not know where the answer to their question will come from, nor is it always obvious whether their question is even answerable. 3 data collection procedure question elicitation: human annotators are given short prompts consisting of the first 100 characters of wikipedia articles and asked to write questions that (a) they are actually interested in knowing the answer to, and (b) that are not answered by the prompt (see section 3.1 for the importance of unseen answers). the prompts are provided merely as inspiration to generate questions on a wide variety of topics; annotators are encouraged to ask questions that are only vaguely related to the prompt. for example, given the prompt apple is a fruit. . . , an annotator might write what disease did steve jobs die of? we believe this stimulation of curiosity reflects how questions arise naturally: people encounter a stimulus such as a scene in a movie, a dog on the street, or an exhibit in a museum and their curiosity results in a question. our question elicitation process is similar to quac in that question writers see only a small snippet of wikipedia content. however, quac annotators were requested to ask about a particular entity while tydi qa annotators were encouraged to ask about anything interesting that came to mind, no matter how unrelated. this allows the question writers even more freedom to ask about topics that truly interest them, including topics not covered by the prompt article. article retrieval: a wikipedia article foot_4 is then paired with each question by performing a google search on the question text, restricted to the wikipedia domain for each language, and selecting the top-ranked result. to enable future use cases, article text is drawn from an atomic wikipedia snapshot of each language. 6 answer labeling: finally, annotators are presented with the question/article pair and asked first to select the best passage answer-a paragraph foot_6 in the article that contains an answer-or else indicate that no answer is possible (or that no single passage is a satisfactory answer). if such a passage is found, annotators are asked to select, if possible, a minimal answer: a character span that is as short as possible while still forming a satisfactory answer to the question; ideally, these are 1-3 words long, but in some cases can span most of a sentence (e.g., for definitions such as what is an atom?). if the question is asking for a boolean answer, the annotator selects either yes or no. if no such minimal answer is possible, then the annotators indicate this. ## typological diversity our primary criterion for including languages in this dataset is typological diversity-that is, the degree to which they express meanings using different linguistic devices, which we discuss below. in other words, we seek to include not just many languages, but many language families. furthermore, we select languages that have diverse data characteristics that are relevant to modeling. for example, some languages may have very little monolingual data. there are many languages with very little parallel translation data and for which there is little economic incentive to produce a large amount of expensive parallel data in the near future. approaches that rely too heavily on the availability of high-quality machine translation will fail to generalize across the world's languages. for this reason, we select some languages that have parallel training data (e.g., japanese, arabic) and some that have very little parallel training data (e.g., bengali, kiswahili). despite the much greater difficulties involved in collecting data in these languages, we expect that their diversity will allow researchers to make more reliable conclusions about how well their models will generalize across languages. ## a quantitative analysis at a glance, tydi qa consists of 204k examples: 166k are one-way annotated, to be used for 19 an analytic language uses helper words rather than morphology to express grammatical relationships. ## recommendations and future work we foresee several research directions where this data will allow the research community to push new boundaries, including: we also believe that a deeper understanding of the data itself will be key and we encourage further linguistic analyses of the data. such insights will help us understand what modeling techniques will be better-suited to tackling the full variety of phenomena observed in the world's languages. we recognize that no single effort will be sufficient to cover the world's languages, and so we invite others to create compatible datasets for other languages; the universal dependency treebank @xcite now has over 70 languages, demonstrating what the community is capable of with broad effort. 26 finally, we note that the content required to answer questions often has simply not been written down in many languages. for these languages, we are paradoxically faced with the prospect that cross-language answer retrieval and translation are necessary, yet low-resource languages will also lack (and will likely continue to lack) the parallel data needed for trustworthy translation systems. ## conclusion confidently making progress on multilingual models requires challenging, trustworthy evaluations. we have argued that question answering is well suited for this purpose and that by targeting a typologically diverse set of languages, progress on the tydi qa dataset is more likely to generalize on the breadth of linguistic phenomena found throughout the world's languages. by avoiding data collection procedures reliant on translation and multilingual modeling, we greatly mitigate the risk of sampling bias. we look forward to the many ways the research community finds to improve the quality of multilingual models.