Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering (2022)

Abstract: We introduce Mintaka, a complex, natural, and multilingual dataset designed for experimenting with end-to-end question-answering models. Mintaka is composed of 20,000 question-answer pairs collected in English, annotated with Wikidata entities, and translated into Arabic, French, German, Hindi, Italian, Japanese, Portuguese, and Spanish for a total of 180,000 samples. Mintaka includes 8 types of complex questions, including superlative, intersection, and multi-hop questions, which were naturally elicited from crowd workers. We run baselines over Mintaka, the best of which achieves 38% hits@1 in English and 31% hits@1 multilingually, showing that existing models have room for improvement. We release Mintaka at https://github.com/amazon-research/mintaka .

## introduction question answering (qa) is the task of learning to predict answers to questions. approaches to question answering include knowledge graph (kg) based methods, which use structured data to find the correct answer @xcite ; machine reading comprehension methods, which extract answers from input documents @xcite ; open domain methods, which learn to retrieve relevant documents and extract or generate answers @xcite , and closed-book methods, which use knowledge implicitly stored in model parameters to answer questions @xcite . with state-of-the-art techniques, qa models can achieve high performance on simple questions @xcite that require a single fact lookup in either a knowledge graph or a text document (e.g., "where was natalie portman born?"). however not all questions are simple in real-world applications. we define complex questions @xcite as questions that require an operation beyond a single fact lookup, such as multi-hop, comparative, or set intersection questions. for example, "what movie had a higher budget, titanic or men in black?" requires looking up the budget of two movies, comparing the values, and selecting the movie with the higher budget. handling more complex questions remains an open problem. one of the challenges in measuring and improving qa performance on complex questions is a lack of datasets. while several qa datasets exist, they have shortcomings of being either large but simple, such as simplequestions @xcite , or complex but small, such as complexquestions @xcite or qald @xcite . recently, several large and complex datasets have been released, including kqa pro @xcite and grailqa @xcite . these datasets use automatically generated questions followed by human paraphrasing, which can result in less natural questions, such as "is the woeid of tuscaloosa 14605?" (kqa pro) or "1520.0 is the minimum width for which size rail gauge?" (grailqa). this can lead to a mismatch between training data and real-world use cases of qa models. in order to address these gaps, we release mintaka, a large, complex, naturally-elicited, and multilingual question answering dataset. mintaka contains 20,000 question-answer pairs elicited in english from crowd workers. we link mintaka to a knowledge graph by asking crowd workers to annotate the question and answer text with wikidata ids. professional translators translated the 20,000 english questions into arabic, french, german, hindi, italian, japanese, portuguese, and spanish, creating a total dataset size of 180,000 questions. in this paper, we present an overview of mintaka in §3, explain how we built mintaka in §4, provide a statistical analysis of the dataset in §5, including a demographic analysis of our crowd workers in §5.3. finally in §6, we present results of existing baseline models on mintaka, the best of which scores 38% ## related works question answering has no shortage of datasets. datasets for question-answering with reading comprehension, such as squad @xcite or natural questions @xcite are often large, and some are even multilingual, such as xquad @xcite , mlqa @xcite , or tydi qa @xcite . these datasets, however, are not explicitly built to be complex, and the answer is usually found in a single passage of text. hotpotqa @xcite and musique @xcite add complexity to reading comprehension by introducing multi-hop questions where the answer requires reasoning over two documents, but neither of these datasets naturally elicit their questions. hotpotqa pre-selects two wikipedia passages and asks workers to write questions using both passages, and musique composes multi-hop questions from existing single hop questions. drop @xcite is another complex reading comprehension dataset, including complex operations such as addition, counting, and sorting. again, drop asks crowd workers to write questions about a selected wikipedia passage. drop additionally introduces a constraint where workers need to write questions that can't be solved by an existing model. within knowledge graph-based question answering (kgqa), webquestionssp @xcite and complexquestions @xcite are more natural qa datasets. both collected real user questions using search query logs or the google suggest api. the answers were annotated manually using freebase as a knowledge graph. webquestionssp contains mostly simple questions, but complexquestions is more complex, including multi-hop questions, temporal constraints, and aggregations. the main drawback of these datasets is size. webquestionssp contains 5k qa pairs, while complexquestions contains only 2k. complexwebquestions @xcite is a dataset based on webquestionssp, which increases the size to 35k qa pairs and introduces more complex operations, including multi-hop, comparatives, and superlatives. however com-plexwebquestions loses some naturalness, as the dataset is built by automatically generating queries and questions, and then asking crowd workers to paraphrase the generated questions. recently, several larger-scale complex kgqa datasets have been released. lc-quad 2.0 @xcite includes 30k questions, including multi-hop questions, and uses the more up-to-date wikidata and dbpedia knowledge graphs. grailqa @xcite is even larger at 64k questions based on freebase with complex questions, including multi-hop, count, and comparatives. kqa pro @xcite is even larger still with 120k questions based on wikidata and with complex questions, including intersection and superlatives. all these datasets make the trade-off of scale over naturalness. to collect question-answer pairs, the authors generate queries from a knowledge graph, generate questions based on the queries, and then ask crowd workers to paraphrase the questions. finally, most datasets are only in english. multilingual and complex datasets are rare. qald 2018 @xcite is one multilingual and complex dataset including 11 languages and complex operations such as counts and comparatives, however includes only 400 questions. by building mintaka, we hope to address an important gap in existing datasets. mintaka questionanswer pairs are both complex and naturallyelicited from crowd workers with no restrictions on what facts or articles the questions can be about. we also translate mintaka into 8 languages, making it one of the first large-scale complex and multilingual question answering datasets. a comparison of mintaka to existing datasets can be seen in ## mintaka mintaka is a complex question answering dataset of 20,000 questions collected in english and translated into 8 languages, for a total of 180,000 questions. mintaka contains question-answer pairs written by crowd workers and annotated with wikidata entities in both the question and answer. we collected questions in eight topics, which were chosen for being broadly appealing and suitable for writing complex questions: movies, music, sports, books, geography, poli-tics, video games, and history. since we want mintaka to be a complex question answering dataset, we explicitly collected questions in the following complexity types: (note: all examples below are from the mintaka dataset.) we also collected translations of the 20,000 english questions in 8 languages using professional translators. since all questions were collected in english from u.s. workers, the questions may have a u.s. bias in terms of the entities (for example, u.s. politicians or books written in english). this is a choice we make since it allows us to create a fully parallel dataset where models can be easily compared across languages. this choice was also made in previous qa datasets @xcite @xcite . ## dataset collection to build our dataset, we used amazon mechanical turk (mturk) in three different tasks. all of our mturk workers were located in the united states, and to ensure high quality, we required workers have an approval rating of 98% and at least 5,000 approved tasks. each of our tasks are explained in the sections below, and examples of the interfaces can be seen in appendix a. ## conclusions in this paper, we introduce mintaka, an end-toend question answering dataset linked to wikidata. mintaka addresses an important gap in qa datasets by being large-scale, complex, naturally-elicited, and multilingual. our baselines show that there is room for improvement in existing methods to handle complex questions, especially in all languages. with the release of mintaka, we hope to encourage researchers to continue pushing the boundaries of question answering to handle more complex questions in more languages. figures 2, 3, and 4 show the interfaces used by mturk workers to complete each of the tasks to build the mintaka dataset. all of these tasks were hosted on mturk. figure 2 is the question elicitation task. this example is for writing questions about the topic movies and the complexity type comparative. in each task, a worker would be shown examples and asked to write five questions. figure 3 is the answer entity annotation task. in this example, a worker is shown the questionanswer pair "q: what oscars did argo win? a: best picture, best adapted screenplay, best film editing" and asked to identify the entities in the answer. the question is given as context, allowing the worker to know that these awards refer to oscar awards. after highlighting each entity, the "search wikidata" button is automatically populated to create a search link on wikidata for the given string that will open in a new window or tab. the workers could then look at all the choices and enter the url of the correct entity. figure 4 is the question entity annotation task. this example again shows the question "what oscars did argo win?" with "oscars" highlighted. workers were asked to focus on one entity at a time, so even though "argo" is also a valid entity in this question, for this task, we are only interested in linking "oscars". workers would first verify that oscars is a valid and complete entity, or modify the string if there was an error. then, similar to the answer entity task, the "search wikidata" button lets worker search the string on wikidata and find the url of the correct entity.