G ame QA : Gamified Mobile App Platform for Building Multiple-Domain Question-Answering Datasets (2023)

Abstract: The methods used to create many of the well-known Question-Answering (QA) datasets are hard to replicate for low-resource languages. A commonality amongst these methods is hiring annotators to source answers from the internet by querying a single answer source, such as Wikipedia. Applying these methods for low-resource languages can be problematic since there is no single large answer source for these languages. Consequently, this can result in a high ratio of unanswered questions, since the amount of information in any single source is limited. To address this problem, we developed a novel crowd-sourcing platform to gather multiple-domain QA data for low-resource languages. Our platform, which consists of a mobile app and a web API, gamifies the data collection process. We successfully released the app for Icelandic (a low-resource language with about 350,000 native speakers) to build a dataset which rivals large QA datasets for high-resource languages both in terms of size and ratio of answered questions. We have made the platform open source with instructions on how to localize and deploy it to gather data for other low-resource languages.

## introduction replicating well known question-answering (qa) data collection methods, such as those used to create the squad @xcite and tydi @xcite datasets, for low-resource languages poses a few problems. first, many large-scale qa datasets are gathered using a single source for answers, e.g. wikipedia. this is problematic since low-resource languages do not have access to any single, large knowledge base from which information can be extracted to create such a dataset. second, qa datasets gathered in an information-seeking manner (where the question is asked prior to finding the answer), will have questions that cannot be answered by the answer source(s). as we will show, the ratio of answerable questions is positively correlated with the amount of content in an answer source. third, many of these methods rely on paid workers to perform the laborious task of annotating data and the necessary funds may not be available in regions of low-resource languages. in this paper, we introduce gameqa, a crowdsourcing platform to build qa datasets. gameqa consists of a mobile trivia app (for ios and android) and a web api. gameqa, which is open source, is specifically designed to gather qa data for low-resource languages. it can be trivially localized and published for specific geographical regions. the main contributions of gameqa are as follows: to spur qa research for low-resource languages, we have made the gameqa platform open source with instructions on how to localize it and subsequently release it for any geographical area 2 . ## related work in recent years, the literature has seen an explosion in the number and diversity of qa datasets @xcite . the most prevalent type of qa datasets are sentence classification and span-prediction datasets. these include documents, questions, and demarcated answer spans that a machine learning model must learn to predict for a given question. @xcite introduced squad, one of the first large spanprediction datasets. they crowd-sourced the creation of the dataset by showing crowd-workers an excerpt from wikipedia and tasking them with writing a question whose answer is contained within the excerpt. this results in qa data with a high lexical overlap between questions and answer paragraphs @xcite which can lead to biased data @xcite . in this paper, we will refer to span-prediction datasets constructed in this manner as being squad-like. coqa @xcite , narrativeqa @xcite , and newsqa @xcite are other examples of datasets falling into this category. to address the problems associated with squadlike datasets, researchers have developed ways that encourage information-seeking behavior during question elicitation. the aim is to emulate human curiosity by having annotators ask questions about something they do not know the answer to. wikiqa @xcite , which poses the problem as a sentence classification problem instead of span prediction, is orders of magnitude smaller than squad (3,047 vs. 100,000 questions, respectively). however, wikiqa brings forth interesting ideas, such as collecting qa data in an information-seeking manner and using websearch queries as a means to capture the curiosity of information-seeking users. this web-search query-based approach was later adopted by larger information-seeking qa datasets, such as ms-marco @xcite , searchqa (dunn et al., 2 https://github.com/cadia-lvl/gameqa 2017), and natural questions @xcite . tydi is an example of an informationseeking dataset constructed using answer paragraphs from the encyclopedic domain @xcite . clark et al. showed crowd-workers a paragraph from wikipedia, but instructed them to ask a question that was not answerable by the paragraph. just as qa datasets can differ in terms of how they source their questions (e.g., squad-like or information-seeking), they can also be categorized in terms of where their answer paragraphs are sourced, i.e. the domain that contains the answers. a very common practice is to constrain a dataset to a single domain -this is the case for the majority of over 80 qa datasets reviewed by @xcite . squad, wikiqa, tydi, and natural questions are examples of such single-source datasets, i.e. they all source answer paragraphs from wikipedia only. examples of datasets sourcing answers from another notable domain, the news domain, are newsqa and cnn/daily mail @xcite . however, low-resource languages are unlikely to have access to a single source that contains enough information to construct a large qa dataset. multiple-domain qa datasets have also been created. ms-marco, which utilized bing foot_1 search queries, used a proprietary state-of-the-art passage retrieval system at bing to match queries with answer paragraphs on the internet. since ms-marco relies on such an algorithm, replicating their methods (i.e. for other languages) is impossible. mmqa, a multiple-domain, squad-like dataset in english and hindi, was created by webcrawling and subsequently having annotators write questions about the crawled articles @xcite . however, it is likely that it suffers from the same problems as other squad-like datasets. to the best of our knowledge, there exists no easily reproducible method in the literature to gather a multi-domain dataset where the questions reflect information-seeking intent. ## the gameqa platform our crowd-sourcing platform consists of a mobile app and web api. the mobile app was written in react native, the web server in node.js, and the underlying database is mongodb. we recruited users by sending an email to all students at reykjavik university and by advertising the app on social media platforms. the app was distributed through apple app store (ios version) and google play store (android version), and, in both cases, only made accessible in iceland. the users form a community where they help each other finding answers to user generated questions. for example, the app might ask a user to write a question. later on, another user would be tasked with reviewing it. once it passes peer review, a third user would be tasked with finding a specific paragraph on a web page containing the answer, using an integrated web-search interface. lastly, another user would verify the answer. these tasks are served randomly and users are not able to review their own content. ## gamification and social features gathering and annotating qa data is a laborious and repetitive task. since the crowd-workers of gameqa are not financially compensated for their contributions, and thus have little incentive to participate, gameqa leverages gamification to incentivize the users and to give them positive feedback when they contribute to the data collection. ## data collection steps in total, the data collection consists of five different stages that each qa pair has to pass: 1) question elicitation, 2) question review, 3) web search and answer paragraph selection, 4) answer span marking, and 5) answer review foot_4 . tasks are randomly served to users -subsequent tasks are thus independent of one another. for our icelandic qa dataset, which we call ruquad (reykjavik university question answering dataset) foot_5 , we sourced answers from five sources in four separate domains: the icelandic wikipedia foot_6 , "vísindavefurinn" (the icelandic web of science) foot_7 , the news websites mbl.is and visir.is, and "stjórnarráðið" (the icelandic government information website) foot_8 . ## results and data analysis throughout our qa collection process for icelandic using gameqa, 1,524 users created an account. roughly 2 3 of those (1,024 users) contributed content to the creation of the ruquad dataset. by the end, they had generated 23,036 questions, 20,730 (90%) of which passed the double peer review. 12,772 answers were annotated and reviewed, resulting in an unanswerable ratio of 38.4%. a preliminary analysis suggests that approximately 30% of the questions that either failed the peer review or were marked as unanswerable might have been mislabeled as such. as a result, the unanswerable rate might become considerably lower with additional labeling after crowd-sourcing the data. there is a remarkable diversity in the number of answer articles. 7,835 articles were gathered in total for the 12,722 answers, i.e. 1.64 answers per article. this ratio is roughly 2.05 and 200 for tydi and squad, respectively. we expect that more diverse answer paragraphs will help a machine learn-ing model, trained on the data, to generalize better. the distribution of articles over the sources is as follows: 68.3% came from the icelandic wikipedia, 18.4% from the icelandic web of science, 13.1% from the two news websites, and 0.2% from the government information website. ## conclusion in this paper, we have presented gameqa -a novel mobile trivia game platform for collecting qa data for low-resource languages. we successfully gamified the experience to increase the number of annotations tasks performed per user and conducted different iterations of user experience testing. the qa data gathered by gameqa's users is culturally relevant for the language and/or geographical region in question. we have made gameqa open source, with instructions on how to localize and subsequently release it for particular geographical areas. we believe that our platform can help to reduce the cost and time associated with collecting qa data for low-resource languages. our method opens up new areas of research e.g. comparing different prompting methods, such as image vs. text prompts, as well as possible advancements for qa research in languages where traditional methods might fail to gather a large-scale qa dataset. given the success of the gamification for collecting qa data, we propose that gamified crowdsourcing can be leveraged to gather data for other nlp tasks as well. for an app such as gameqa, there is a target user base that is interested in triva and knowledge and thus willing to annotate data in this manner. similarly, for other nlp tasks, such as machine translation, there exists a potential user base of multilingual persons that are greatly interested in languages and translation. we see great potential in applying the knowledge learned through implementing gameqa for such tasks.