A Benchmark Suite of J apanese Natural Questions (2024)

Abstract: To develop high-performance and robust natural language processing (NLP) models, it is important to have various question answering (QA) datasets to train, evaluate, and analyze them. Although there are various QA datasets available in English, there are only a few QA datasets in other languages. We focus on Japanese, a language with only a few basic QA datasets, and aim to build a Japanese version of Natural Questions (NQ) consisting of questions that naturally arise from human information needs. We collect natural questions from query logs of a Japanese search engine and build the dataset using crowdsourcing. We construct Japanese Natural Questions (JNQ) and a Japanese version of BoolQ (JBoolQ), which is derived from NQ and consists of yes/no questions. JNQ consists of 16,871 questions, and JBoolQ consists of 6,467 questions. We also define two tasks from JNQ and one from JBoolQ and establish baselines using competitive methods drawn from related literature. We hope that these datasets will facilitate research on QA and NLP models in Japanese. We are planning to release JNQ and JBoolQ.

## introduction to develop high-performance and robust natural language processing (nlp) models, it is important to have various question answering (qa) datasets to train, evaluate, and analyze them. there are diverse extractive and generative qa datasets that require many techniques and knowledge to solve, such as multi-hop inference @xcite and real-world knowledge @xcite . there have been some studies to solve many qa tasks in an integrated manner, rather than solving them individually, such as unified qa @xcite and flan @xcite . however, such an integrated analysis is possible only in english but not in other languages because of the lack of qa datasets. this study focuses on japanese, which has only a few basic qa datasets, such as jsquad @xcite , jaquad @xcite , and jaqket @xcite . in this paper, we focus on natural questions (nq) @xcite , which consist of questions that arise naturally from human information needs, as a critical qa dataset that does not exist in japanese. qa datasets such as squad @xcite have the problem of annotation artifacts @xcite because the questions are manually created by annotators, which are not natural. in contrast, nq uses queries entered by users in a search engine, which are considered natural questions. one possible approach to creating a japanese version of nq is translating the original nq dataset into japanese. however, we do not use translation due to concerns about the unnaturalness of translated sentences, which can result from differences in grammar and other linguistic factors, as well as potential cultural differences between japan and other countries. instead, we build and publish japanese natural questions (jnq) using query logs from a japanese search engine. we also re-define the dataset specification of the original nq to obtain a better nq dataset. @xcite have hired trained annotators to build the nq dataset, but for jnq, we use crowdsourcing to reduce costs. this method can be applied to any language in which search engine query logs are available. in addition to jnq, we build jboolq, a japanese version of boolq @xcite . boolq is derived from nq and consists of yes/no questions. jboolq questions and yes/no answers are collected in the same way as jnq. in the original boolq, there are only two options: "yes" or "no". however, to make the setting more realistic, we add an option of "unable to answer" to jboolq, represented as "none". this makes our dataset more challenging than the original boolq. in consequence, jnq contains 16,871 queries and 80,288 paragraphs. jboolq, combined with the jnq yes/no questions, contains 6,467 queries and 31,677 paragraphs. examples of jnq and jboolq are shown in figure 1 . furthermore, we define three tasks using the two datasets as a new qa benchmark in japanese: long answer extraction, short answer extraction, and yes/no answer identification (boolq). we also evaluate these tasks with their respective baselines. jnq and jboolq will be available online. ## related work existing qa datasets can be broadly categorized into those where the questions are natural and those where they are not. qa datasets where the questions are not natural mainly include squad @xcite and squad 2.0 @xcite . the questions in these datasets are not natural because annotators create them after reading a paragraph. therefore, annotation artifacts in the created questions and lexical overlap between questions and paragraphs are problematic when using these datasets. natural questions @xcite ) and boolq @xcite are qa datasets that contain natural questions. to build these datasets, search engine query logs are used to collect natural questions arising from human information needs. the documents are wikipedia articles, and the answers consist of long answers (e.g., paragraphs or tables) and short answers (spans or yes/no). other datasets that collect questions from query logs include wikiqa @xcite and ms marco @xcite . in these datasets, the answer format differs from nq and boolq, with a single sentence in the document or a hand-crafted summary. qa datasets whose questions are not derived from query logs but are claimed to be natural include tydi qa @xcite , icelandic nq @xcite , and russian boolq @xcite . in these datasets, annotators are given a prompt consisting of a part or summary of a document and asked to think of a question that cannot be answered by reading only the prompt. these questions are claimed to be "natural" because they are derived from what humans wanted to know about the prompt. however, they are not naturally occurring questions because the authors ask them to think of a question. thus, we consider that they are not truly natural questions. for non-english qa datasets, there are several multilingual qa datasets, such as tydi qa @xcite , mlqa @xcite , xor qa @xcite , and xquad @xcite . however, only approximately half of them include japanese. due to the lack of diverse datasets in japanese, we construct japanese natural questions from scratch. japanese natural questions (jnq), like nq, consists of quadruples of a question, a document, long answer(s), and short answer(s). the questions are extracted from search engine query logs, and the documents are japanese wikipedia articles. the long answers and short answers are obtained using crowdsourcing. by using crowdsourcing, it is possible to construct a dataset at a low cost and with some quality level without expert annotators. we limit the long answers only to paragraphs to simplify the task, considering that dataset construction is conducted using crowdsourcing. although nq has a strict restriction that there is at most one long answer in a document, there are often multiple paragraphs containing answers. therefore, jnq allows for scenarios with multiple long answers to a single question. we describe each stage of building jnq below. in crowdsourcing, 10 crowdworkers are assigned to deal with a task to build a high-quality benchmark. in cases where ambiguity is detected due to diverging opinions among crowdworkers at each stage, such instances are not incorporated into jnq. we illustrate the construction flow in figure 2 . ## japanese boolq boolq @xcite is a qa dataset focusing on natural yes/no questions. it contains many non-factoid questions that require a wide range of inferential abilities to answer. each instance consists of a question, a paragraph (equivalent to a long answer in nq), and an answer (yes/no). the questions and paragraphs are extracted from search engine query logs and wikipedia articles, like nq. boolq adopts only the questions with either yes or no answers and pairs them with not a whole document but a paragraph to simplify the specification. japanese boolq (jboolq) consists of a question, a document, a long answer, and a yes/no answer, like yes/no questions in jnq. unlike boolq, each question may have multiple long answers, and the answers can include "none", which means unanswerable, in addition to yes/no. therefore, it is more challenging than boolq, and a deeper understanding of the documents is required to answer the questions. we construct jboolq using basically the same procedure as jnq. since the ratio of yes/no questions in jnq is only around 1%, for jboolq, we collect questions from a larger query log pool than jnq. the construction procedure is as follows. the details of each step are described in section 3. 1. question and document collection foot_3 2. good question identification 3. yes/no question identification 4. long answer identification 5. yes/no answer identification compared to jnq, the order of yes/no question identification and long answer identification is reversed to narrow down the candidates to the target yes/no questions at an early stage and reduce the annotation cost later. finally, we merge the yes/no questions in jnq into jboolq. ## analysis in this section, we analyze jnq and jboolq. ## experiments 6.1 experimental setup we define three tasks to use jnq and jboolq as a benchmark for evaluating qa systems. from jnq, we introduce the following two tasks: long answer extraction, short answer extraction. from jboolq, we introduce the task of yes/no answer identification. we also establish baselines using competitive methods drawn from related literature. we implement hyperparameter searches and report the best scores. we list the statistics of the tasks in long answer extraction unlike nq, in our dataset, there can be multiple long answers or no long answer in a document. thus, we consider long answer extraction as a paragraph-based multilabel classification task. given a question and a document, a system tries to select all paragraphs with long answers. we use precision, recall, and f1 scores for evaluation metrics. we introduce a baseline that considers the task a binary classification problem. for each paragraph in the document, we input the question-paragraph pair into the model and binarily decide whether the paragraph is a long answer. we use japanese bert @xcite and roberta @xcite as base models foot_4 . we use two kinds of training sets in our experiments: (1) the paragraphs collected in section 3.3, which contain positive examples and hard negative examples (challenge candidates, which have high relevance to the snippet but are considered as no long answer), and (2) all paragraphs in the documents. the ambiguous paragraphs are excluded from both. for testing, we use all paragraphs in the documents, aiming to be close to real extraction scenarios. we also evaluate human performance using crowdsourcing in the same way as the dataset construction process. we asked 10 annotators to answer. if seven or more annotators agree, it is considered that the paragraph is a long answer; otherwise, it is not. due to cost reasons, we sampled 100 questions for human evaluation instead of using the whole test set. ## conclusion we constructed two qa datasets: japanese natural questions (jnq) and japanese boolq (jboolq). the questions in these datasets are collected from query logs from a japanese search engine and are natural, derived from human information needs. the annotation process was conducted through crowdsourcing. we also defined a total of three tasks, including long answer extraction, short answer extraction, and yes/no answer identification. we evaluated the performance of the baseline models. the constructed datasets can be used for training, evaluating, and analyzing qa and nlp models and are expected to facilitate these studies in japanese.