A Relation Extraction Dataset for Knowledge Extraction from Web Tables (2022)

Abstract: Relational web-tables are significant sources of structural information that are widely used for relation extraction and population of facts into knowledge graphs. To transform the web-table data into knowledge, we need to identify the relations that exist between column pairs. Currently, there are only a handful of publicly available datasets with relations annotated against natural web-tables. Most datasets are constructed using synthetic tables that lack valuable metadata information, or are limited in size to be considered as a challenging evaluation set. In this paper, we present REDTab, the largest natural-table relation extraction dataset. We have annotated ~9K tables and ~22K column pairs using crowd sourced annotators from MTurk, which has 50x larger number of column pairs than the existing human-annotated benchmark. Our test set is specially designed to be challenging as observed in our experiment results using TaBERT. We publicly release REDTab as a benchmark for the evaluation process in relation extraction.

## introduction web-tables contain a lot of knowledge @xcite that can be utilized for various tasks such as question answering @xcite , building knowledge graphs @xcite , web-search @xcite , data-mining @xcite and so on. relation extraction (re) is one way to extract table information by capturing relations between two table columns (figure 1 ). however, the current re datasets available are either synthetic, or very small in size. in this paper, we propose a new human-annotated dataset for the evaluation of re models. the existing benchmark datasets for column pair re are t2dv2 and semtab. however, t2dv2 @xcite only contains 236 tables and 435 column pairs annotated with 118 relations where about 50% of relations have only 1 column pair sample. the highest number of samples for a relation is only 28. hence, due to its small size and less diversity in column pairs per relation, it is difficult to be treated as a reliable evaluation set. in contrast, the semtab 2020 challenge @xcite provides a large benchmark dataset, but synthetically generated, which is arguably not representative of the real web-tables. other table-based datasets @xcite @xcite are designed for a binary relation extraction task. redtab foot_0 is a human annotated dataset consisting of 9,149 tables and 22,236 column pairs with 23 relations. the table was collected from web data commons (wdc) @xcite relational tables for music and literature domains. these domains are two of most popular domains for question answering @xcite @xcite . the dataset also contains metadata such as table title, page title, and text before and after the table as extra context. the key differences between our proposed dataset and other benchmarks are shown in ## data collection and annotation in this section, the construction of the redtab dataset is described in detail. the discussion is divided into three subsections: data source, annotation protocol and dataset validation. ## experimental setup we use tabert @xcite as a baseline in our dataset. tabert is a pre-trained language model that jointly learns representations for natural language sentences and tables. tabert represents each cell by the column name, column datatype and cell value, which is followed by using the transformer @xcite id, and context. we feed the table header and row data as header and data respectively. the table title is assigned as the tabert id, while the page title is assigned as the tabert context. we then concatenate the column embeddings for the current input, i.e. concat(embedding of the left column, embedding of the right column). this is connected with a classification layer to predict the relation (see figure 4 ). we experiment on both base and large tabert model. our model is fine-tuned on 4 gpus with a batch-size of 16 for 3 epochs. the model is trained with adamw optimizer @xcite . each experiment takes less than an hour to finish. ## related work we briefly describe most major benchmark datasets that contain relation annotations for web-tables. ## conclusion we present redtab, the largest natural web-tables dataset for column pair relation extraction. the dataset is annotated by crowd sourced annotators from mturk. redtab includes more than 9k tables and 22k column pairs making it the largest human-annotated relation extraction dataset to our knowledge. we evaluate state-of-the art table embedding model tabert and find that our dataset is challenging over the strategically created train-test split set. our final set contains more diverse set of column pairs per relation that is ideal for testing models. furthermore, our analysis shows that predicting relations solely based on table row data is challenging and use of some metadata information significantly improves the performance. we expect this dataset can contribute to facilitate further progress in the field of information extraction research.