Developing a How-to Tip Machine Comprehension Dataset and its Evaluation in Machine Comprehension by BERT (2020)

Abstract: In the field of factoid question answering (QA), it is known that the state-of-the-art technology has achieved an accuracy comparable to that of humans in a certain benchmark challenge. On the other hand, in the area of non-factoid QA, there is still a limited number of datasets for training QA models, i.e., machine comprehension models. Considering such a situation within the field of the non-factoid QA, this paper aims to develop a dataset for training Japanese how-to tip QA models. This paper applies one of the state-of-the-art machine comprehension models to the Japanese how-to tip QA dataset. The trained how-to tip QA model is also compared with a factoid QA model trained with a Japanese factoid QA dataset. Evaluation results revealed that the how-to tip machine comprehension performance was almost comparative with that of the factoid machine comprehension even with the training data size reduced to around 4% of the factoid machine comprehension. Thus, the how-to tip machine comprehension task requires much less training data compared with the factoid machine comprehension task.

## introduction recent advances in the field of qa or machine comprehension are mostly in the domain of factoid qa related to wikipedia articles and news articles @xcite @xcite . one of the most well-known qa datasets and benchmark tests is the stanford question answering dataset (squad) @xcite , which is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of wikipedia articles, where the answer to every question is a text segment, or span, from the corresponding reading passage, or the question might be unanswerable. it is reported foot_0 that state-of-theart machine comprehension models trained with squad outperform humans @xcite . however, apart from the issues related to developing benchmark datasets for factoid qa and improving state-of-the-art general-purpose machine comprehension models, there is a relatively limited number of published literature that handles issues, such as the development of datasets for nonfactoid qa and the application of state-of-the-art general-purpose machine comprehension models to those non-factoid datasets. typical non-factoid qa tasks include opinion qa, definition qa, reason qa, and how-to tip qa. among various kinds of non-factoid knowledge which are the key to developing techniques for non-factoid qa tasks, a recent study @xcite examined the types of japanese websites which include various howto tips related to job hunting, marriage, and apartment. the study @xcite also aims to automatically identify those how-to tip websites, which will be an important knowledge source for training how-to tip qa models. considering such a situation, within the field of nonfactoid qa, this paper studies how to develop a dataset for training japanese how-to tip (hereafter throughout the paper, we use the simplified term "tip") qa models. as examples in this paper, we developed tip qa datasets for 'job hunting," "marriage," "apartment," "hay fever," "dentist," and "food poisoning," where "job hunting" and "marriage" tip qas are for both training and testing, while other tip qas are only for testing. for "job hunting", figure 1 presents a typical example of a tuple of a context, a tip question, and an answer. furthermore, in order to understand rough idea of figure 1 : an example of the machine comprehension model of tip qa for "job hunting" together with an example of a tuple of a context c, a question q, and answer a (extracted from a column web page entitled "formatting tips for your curriculum vitae (cv)" ( https://www.thebalancecareers.com/ curriculum-vitae-format-2060351) from a tip website titled "the balance careers" ( https://www.thebalancecareers.com/ ) ) the (how-to) tip questions we study in this paper in the broader sense, we manually classify them into five types as shown in this paper then applies bert @xcite , one of the state-of-the-art machine comprehension models, to a japanese tip qa dataset. the trained tip qa model is also compared with a factoid qa model which is also trained with a japanese factoid qa dataset. evaluation results revealed that the tip machine comprehension performance was almost comparative with that of the factoid machine comprehension even with the training data size reduced to around 4% of the factoid machine comprehension. thus, the tip machine comprehension task requires much less training data compared with the factoid machine comprehension task. ## query focuses and collecting web pages this section briefly describes the workflow of collecting web pages. first, the notion of query focus is a keyword used for every search request related to a specific subject. for example, whenever the aim was to collect web pages about anything related to job hunting, the word "job hunting" was always put at the beginning of the query, and all available suggested keywords provided by the search engine were collected, such as "job hunting self-promotion" and "job hunting portfolio." using all such suggested keywords as queries (called search engine suggests or suggests), the search engine is crawled, and top 10 results for each suggest are collected. ## conclusion this paper explored a way to develop a dataset for training japanese tip qa models, and it applied bert @xcite to a japanese tip qa dataset. evaluation results revealed that the tip machine comprehension performance was almost comparative with that of the factoid machine comprehension even with the training data size reduced to around 4% of the factoid machine comprehension. thus, the tip machine comprehension task requires much less training data compared with the factoid machine comprehension task. future direction of this work includes applying the proposed framework of tip machine comprehension to other languages, such as english and chinese. in both languages, factoid qa datasets are publicly available (e.g., squad @xcite for english and cmrc2018 @xcite for chinese), and it is quite attainable to train a factoid machine comprehension model by fine-tuning the bert pre-trained model and then to directly apply the factoid machine comprehension model to the tip machine comprehension task. actually, as a preliminary work, a chinese factoid machine comprehension model is trained by finetuning the pre-trained multilingual cased model with cmrc2018 chinese factoid qa dataset 1011 , and then applying it to 30 chinese tip questions on "marriage" with context texts. as a result, around 50% accuracy for manual evaluation is achieved by exact and partial match, which is almost comparative to the performance achieved in the japanese tip machine comprehension task reported in this paper. thus, it is expected that extending the proposed framework of tip machine comprehension to other languages, such as english and chinese, is quite straightforward. another future direction is to extending the proposed framework of tip machine comprehension to open domain tip machine comprehension. this extension is similar to the extension of existing factoid machine comprehension with wikipedia texts' paragraphs as contexts to open domain machine comprehension with the whole wikipedia articles @xcite . in the extended open domain tip machine comprehension framework, the document retriever module is realized based on the tip websites search and column web page collection architectures proposed in this paper. the document reader module can be easily realized by simply applying the tip machine comprehension model of this paper. another definitely important future direction should be to invent a technique of how to automate the procedure of collecting column web pages and generating the tuple of a context c, a question q, and answer a. this task can be regarded as that of training a tip machine comprehension model from a noisy training dataset.