From Dataset Recycling to Multi-Property Extraction and Beyond (2020)

Abstract: This paper investigates various Transformer architectures on the WikiReading Information Extraction and Machine Reading Comprehension dataset. The proposed dual-source model outperforms the current state-of-the-art by a large margin. Next, we introduce WikiReading Recycled - a newly developed public dataset, and the task of multiple-property extraction. It uses the same data as WikiReading but does not inherit its predecessor’s identified disadvantages. In addition, we provide a human-annotated test set with diagnostic subsets for a detailed analysis of model performance.

## introduction the emergence of attention-based models has revolutionized natural language processing @xcite . pretraining these models on large corpora like bookcorpus @xcite has been shown to yield a reliable and robust base for downstream tasks. these include natural language inference @xcite , question answering @xcite , named entity recognition @xcite @xcite , and property extraction @xcite . the creation of large supervised datasets often comes with trade-offs, such as one between the quality and quantity of data. for instance, the wikireading dataset @xcite has been created in such a way that wikidata annotations were treated as the expected answers for related wikipedia articles. however, the above datasets were created separately, and the information content of both sources overlaps only partially. hence, the resulting dataset may contain noise. the best models can achieve results better than the human baseline across many nlp datasets such as mscqas @xcite , sts-b, qnli @xcite , cola or mrpc @xcite . however, as a consequence of different kinds of noise in the data, they rarely maximize the score metric @xcite . while current work in nlp is focused on preparing new datasets, we regard recycling the current ones as equally important as creating a new one. thus, after outperforming previous state-of-the-art on wikireading, we investigated the dataset's weaknesses and created an entirely new, more challenging multi-property extraction task with improved data splits and a reliable, human-annotated test set. contribution. the specific contributions of this work are the following. we analyzed the wikireading dataset and pointed out its weaknesses. we introduced a multi-property extraction task by creating a new dataset: wikireading recycled. our dataset contains a human-annotated test set, with multiple subsets aimed to benchmark qualities such as generalization on unseen properties. we introduced a mean-multi-property-f 1 score suited for the new multi-property extraction task. we evaluated previously used architectures on both datasets. furthermore, we showed that pretrained transformer models (dual-source roberta and t5) beat all other baselines. the new dataset and all the models mentioned in the present paper were made publicly available on github. 1 2 related work early work in relation extraction revolves around problems crafted using distant supervision methods, which are semi-supervised methods that automatically label pools of unlabeled data @xcite . in contrast, many qa datasets were created through crowd-sourcing, where annotators were asked to formulate questions with answers that require knowledge retrieval and information synthesis. one of the most popular qa datasets is wikipedia-based squad, where an instance consists of a human-formulated question, and an encyclopedic reading passage used to base the answer on @xcite . another crowd-sourced dataset that profoundly influenced natural language inference research is snli (bowman et al., 2015)-a three-way semantics-based classification of a relation between two different sentences. both squad and snli are large-scale machine reading comprehension (mrc) tasks, but they cannot be treated as property extraction as defined in section 3; hence they are not considered in this paper. similarly, some mrc problems framed in trec tracks, such as conversational assistance or question answering, are beyond the scope of this paper @xcite . @xcite proposed the wikireading dataset that consists of a wikipedia article and related wikidata statement. no additional annotation work was performed, yet the resulting dataset was of presumably high reliability. nevertheless, we consider an additional human annotation to be desired (section 4.3). alongside the dataset, a property extraction task was introduced. the idea behind it is to read an article given a property name and to infer the associated value from the article. the property extraction paradigm is described in detail in section 3, whereas a brief comparison to related datasets is presented in initially, the best-performing model used placeholders to allow rewriting out-of-vocabulary words to the output. next, @xcite presented a reinforcement learning approach that improved results on a challenging subset of the 10% longest articles. this framework was extended by @xcite with a self-correcting action that removes the inaccurate answer from the answer generation module and continues to read. to the best of our knowledge, no authors validated transformer-based models on wikireading and pretrained encoders. ## property extraction let a property denote any query for which a system is expected to return an answer from given text. examples include country of citizenship for a biography provided as an input text, or architect name for an article regarding the opening of a new building. contrary to qa problems, a query is not formulated as a question in natural language but rather as a phrase or keyword. we use the term value when referring to a valid answer for the stated query. some properties have multiple valid answers; thus, multiple values are expected. examine the case of johann sebastian bach's biography for which property sister has eight values. we will refer to any task consisting of a tuple (properties, text) for which values are to be provided as a property extraction task. the biggest publicly available dataset for property extraction is wikireading @xcite . wikipedia with wikidata information. the dataset is of great value; however, several flaws can be identified. first, more than 95% of articles in the test set appeared in the train set ( ## wikireading recycled: novel dataset for multi-property extraction the comparison to existing datasets and shared tasks is briefly presented in ## model architectures we evaluate different model architectures on the wikireading recycled dataset. we reimplemented the previously best performing wikireading model, finetuned pretrained transformer models, and applied a dual-source model. their competitiveness can be demonstrated by the fact that we were able to outperform the previous state-of-the-art on the wikireading by a far margin. ## evaluation in this section, we describe the evaluation of previously proposed architectures on both wikireading and wikireading recycled datasets. we would like to highlight that the results are not comparable between the two datasets, as they are based on different train/validation/test splits. ## discussion and analysis the final scores of transformer-based models differ slightly on wikireading recycled. in order to get more insight, we analyze the models on diagnostic sets described in section 4.4. ## summary we introduced wikireading recycled-the first multi-property extraction dataset with a humanannotated test set. we provided strong baselines that improved the current state-of-the-art on wikireading by a large margin. the bestperforming architecture was successfully adapted from automatic post-editing systems. we show that using pretrained language models increases the performance on the wikireading recycled dataset significantly, despite its large size. additionally, we created diagnostic subsets to qualitatively assess model performance. the results on a challenging subset of unseen properties reveal that despite high overall scores, the evaluated systems fail to provide satisfactory performance. low scores indicate an opportunity to improve, as these properties were verified by annotators and are expected to be answerable. we look forward to seeing models closing this gap and leading to remarkable progress in machine reading comprehension. the dataset and models, as well as their detailed configurations required for reproducibility, are publicly available.