MATINF : A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization (2020)

Abstract: Recently, large-scale datasets have vastly facilitated the development in nearly all domains of Natural Language Processing. However, there is currently no cross-task dataset in NLP, which hinders the development of multi-task learning. We propose MATINF, the first jointly labeled large-scale dataset for classification, question answering and summarization. MATINF contains 1.07 million question-answer pairs with human-labeled categories and user-generated question descriptions. Based on such rich information, MATINF is applicable for three major NLP tasks, including classification, question answering, and summarization. We benchmark existing methods and a novel multi-task baseline over MATINF to inspire further research. Our comprehensive comparison and experiments over MATINF and other datasets demonstrate the merits held by MATINF.

## introduction in recent years, large-scale datasets (e.g., ima-genet @xcite and squad @xcite ) have inspired remarkable progress in many areas like computer vision (cv) and natural language processing (nlp). on the one hand, well-annotated data provide essential information for training supervised machine learning models. on the other hand, benchmarked datasets make it possible to evaluate and compare the capability of different methods on the same stage. due to the high cost of data annotation, existing nlp datasets are usually labeled for only one particular task (e.g., squad @xcite for question answering, cnn/dm @xcite for summarization and agnews @xcite for text classification). these singletask datasets hinder the development of learning common and task-invariant knowledge @xcite . although multi-task learning and transfer learning have delivered encouraging results, we still cannot determine whether the improvement is from the extension of input or supervision. furthermore, task-specific data make the models tend to learn task-specific leakage features @xcite rather than meaningful knowledge that could generalize to other tasks. however, as a key step to artificial general intelligence (agi), knowledge acquisition requires the model to learn more general knowledge instead of overfitting on a specific task. therefore, a large-scale and cross-task dataset is in huge demand for future nlp research. nevertheless, to the best of our knowledge, none of the existing datasets could meet such demand. in this paper, we propose maternal and infant dataset (matinf), the first large-scale dataset covering three major nlp tasks: text classification, question answering and summarization. matinf consists of question answering data crawled from a large chinese maternity and baby caring qa site. on this site, users can ask questions related to maternity and baby caring. when submitting a question, a detailed description is required to provide essential information and the asker also needs to assign a category for this question from a pre-defined topic list. each user could submit an answer to a question post, and the asker will select the best answer out of all the candidates. to attract more attention, the askers are encouraged to set rewards using virtual coins when submitting the question and these coins will be given to the user who submitted the best answer selected by the asker. this rewarding mechanism could constantly ensure high-quality answers. matinf supports three nlp tasks as follows. text classification. given a question and its detailed description, the task is to select an appropriate category from the fine-grained category list. different from previous news classification tasks whose category set is general topics like entertainment and sports, matinf-c is a fine-grained classification under a single domain. that is, the distance between different categories is smaller, which provides a more challenging stage to test the continuously evolving state-of-the-art neural models. ## matinf dataset we present maternal and infant (matinf) dataset, a large-scale dataset jointly labeled for classification, question answering and summarization in the domain of maternity and baby caring in chinese. an entry in the dataset includes four fields: question (q), description (d), class (c) and answer (a). an example is shown in figure 1 , and the average character and word numbers of each field are reported in we collect nearly two million question-answer pairs with fine-grained human-labeled classes from a large chinese maternity and baby caring qa site. we conduct both automatic and manual data cleansing and remove: (1) classes with insufficient samples; (2) entries in which the length of the description filed is less than the length of the question field; (3) data with any field longer than 256 characters; (4) human-spotted ill-formed data. after the data cleansing, we construct matinf with the remaining 1.07 million entries. we first randomly split the whole data into training, validation and test sets with a proportion of 7:1:2. then, we use the splits for summarization and qa. for classification, we further divide the data into two sub-tasks according to different classification standards within each split. ## multi-task learning recently, many attempts have been made on multitask learning in nlp @xcite @xcite @xcite @xcite @xcite @xcite and several benchmarks are available for multi-task evaluation @xcite . though recent studies show that multi-task learning is effective, there is still one more question to answer. that is, when training models on multiple tasks, multiple datasets are used by default. as illustrated in figure ## experiments in this section, we benchmark a few baselines and mtf-s2s on the three tasks of matinf. we run each experiment with three different random seeds and report the average result of the three runs. ## discussion since matinf is a web-crawled dataset, it would be inevitable to be noisier than a dataset annotated by hired annotators though we have made every effort to clean the data. on the bright side, it can encourage more robust models and facilitate realworld applications. for future work, we would like to see more interesting work exploring new multi-task learning approaches. ## conclusion to conclude, in this paper, we present matinf, a jointly labeled large-scale dataset for classification, question answering and summarization. we benchmark existing methods and a straightforward baseline with a novel multi-task paradigm on mat-inf and analyze their performance on these three tasks. our extensive experiments reveal the potential of the proposed dataset for accelerating the innovations in the three tasks and multi-task learning.