Text-to-Table: A New Way of Information Extraction (2022)

Abstract: We study a new problem setting of information extraction (IE), referred to as text-to-table. In text-to-table, given a text, one creates a table or several tables expressing the main content of the text, while the model is learned from text-table pair data. The problem setting differs from those of the existing methods for IE. First, the extraction can be carried out from long texts to large tables with complex structures. Second, the extraction is entirely data-driven, and there is no need to explicitly define the schemas. As far as we know, there has been no previous work that studies the problem. In this work, we formalize text-to-table as a sequence-to-sequence (seq2seq) problem. We first employ a seq2seq model fine-tuned from a pre-trained language model to perform the task. We also develop a new method within the seq2seq approach, exploiting two additional techniques in table generation: table constraint and table relation embeddings. We consider text-to-table as an inverse problem of the well-studied table-to-text, and make use of four existing table-to-text datasets in our experiments on text-to-table. Experimental results show that the vanilla seq2seq model can outperform the baseline methods of using relation extraction and named entity extraction. The results also show that our method can further boost the performances of the vanilla seq2seq model. We further discuss the main challenges of the proposed task. The code and data are available at https://github.com/shirley-wu/text_to_table .

## introduction information extraction (ie) is a task that aims to extract information of interest from text data and represent the extracted information in a structured form. traditional ie tasks include named entity recognition which recognizes entities and their 1 the work was done when xueqing wu was an intern at bytedance ai lab. figure 1 : an example of text-to-table from the rotowire dataset. the text is a report of a basketball game, and the tables are the scores of the teams and players. in this work, we study ie in a new setting, referred to as text-to-table. first, the system receives a training dataset containing text-table pairs. each text-table pair contains a text and a table (or tables) representing information needed for the target application extracted from the text. the system learns a model for information extraction. next, the system employs the learned model to conduct information extraction from a new text and outputs the result in a table (or tables). figure 1 gives an example of text-to-table, where the input (above) is a report of a basketball game, and the output (below) is two tables summarizing the scores of the teams and players from the input. text-to-table is unique compared to the traditional ie approaches. first, text-to-data can be performed at both sentence-level and documentlevel. while the distinction between sentence and document level is vague, document-level extraction can produce a more complex output. as in the example in figure 1 , extraction of information is performed from the entire document. the extracted information contains multiple types of scores of teams and players in a basketball game structured in table format. second, the schemas for extraction are implicitly included in the training data such as header names. there is no need to explicitly define the schemas, which reduces the need for manual efforts for schema design and annotations. our work is inspired by research on the so-called table-to-text (or data-to-text) problem, which is the task of generating a description for a given table . table -to-text is useful in applications where the content of a table needs to be described in natural language. thus, text-to-table can be regarded as an inverse problem of table-to-text. however, there are also differences. most notably, their applications are different. text-to-table systems can automatically produce tables for text summarization and text mining. for example, the score tables of sports games and infoboxes of wikipedia articles can serve as summaries of original documents. the score tables can be utilized to evaluate the athletes' performances, and the infoboxes can be used to construct a knowledge graph. in this work, we formalize text-to-table as a sequence-to-sequence (seq2seq) task. more specifically, we translate the text into a sequence representation of a table (or tables), where the schema of the table is implicitly contained in the representation. we build the seq2seq model on top of a pre-trained language model, which is the state-ofthe-art approach for seq2seq tasks @xcite . although the approach is a natural application of existing technologies, as far as we know, there has been no previous study to investigate to what extent the approach works. we also develop a new method for text-to- the approach to ie based on seq2seq has already been proposed. methods for conducting individual tasks of relation extraction @xcite @xcite , named entity recognition @xcite , event extraction @xcite and role-filler entity extraction @xcite have been developed. methods for jointly performing multiple tasks of named entity recognition, relation extraction, and event extraction have also been devised @xcite . most of the methods exploit suitable pre-trained models such as bert. however, all the existing methods rely on predefined schemas for extraction. moreover, their models are designed to extract information from short texts, rather than long texts, and extract information with simple structures (such as an entity and its type), rather than information with complicated structures (such as a table ). we conduct extensive experiments on the four datasets. results show that the vanilla seq2seq model fine-tuned from bart @xcite can outperform the state-of-the-art ie models finetuned from bert @xcite . furthermore, results show that our proposed approach to text-to-table with the two techniques can further improve the extraction accuracies. we also summarize the challenging issues with the seq2seq approach to text-to-table for future research. our contributions are summarized as follows: traditionally, researchers formalize the task as a language understanding problem. the state-ofthe-art methods for ner perform the task on the basis of the pre-trained language model bert @xcite . the pipeline approach to re divides the problem into ner and relation classification, and conducts the two sub-tasks in a sequential manner @xcite , while the end-to-end approach jointly carries out the two sub-tasks @xcite @xcite . the state-of-the-art methods for ee also employ bert and usually jointly train the models with other tasks such as ner and re @xcite @xcite . all the methods assume the use of pre-defined schemas (e.g., entity types for ner, entity and relation types for re, and event templates for ee). besides, most methods are designed for extraction from short texts. therefore, existing methods for ie cannot be directly applied to text-to-table. another series of related work is open information extraction (openie), which aims to extract information from texts without relying on explicitly defined schemas @xcite @xcite @xcite . however, openie aims to extract information with simple structures (i.e., relation tuples) from short texts, and the methods in openie cannot be directly applied to text-to-table. ie is also conducted at document level, referred to as doc-level ie. for example, some ner methods directly perform ner on a long document @xcite , and others encode each sentence in a document, use attention to fuse document-level information, and perform ner on each sentence @xcite . there are also re methods that predict the relationships between entities in a document @xcite . however, existing doc-level ie approaches usually do not consider the extraction of complex relations between many items. sequence-to-sequence (seq2seq) is the general problem of transforming one text into another text @xcite , which includes machine translation, text summarization, etc. the use of the pre-trained language models of bart @xcite and t5 (raffel et al., 2020) can significantly boost the perfor-mances of seq2seq, such as machine translation @xcite @xcite ) and text summarization @xcite @xcite . recently, some researchers also formalize the ie problems as seq2seq, that is, transforming the input text into an internal representation. one advantage is that one can employ a single model to extract multiple types of information. results show that this approach works better than or equally well as the traditional approach of language understanding, in re @xcite , ner @xcite , ee @xcite and ree @xcite . methods that jointly perform multiple tasks including ner, re, and ee have also been devised @xcite . data-to-text aims to generate natural language descriptions from the input structured data such as sports commentaries @xcite . the structured data is usually represented as tables @xcite @xcite , sets of table cells @xcite , semantic representations @xcite , or sets of relation triples @xcite . the task requires the model to select the salient information from the data, organize it in a logical order, and generate an accurate and fluent natural language description @xcite . data-to-text models usually adopt the encoder-decoder architecture. the encoders are specifically designed to model the input data, such as multi-layer perceptron @xcite , recurrent neural network @xcite @xcite , graph neural network @xcite , or transformer @xcite . ## problem formulation as shown in figure 1 , text-to-table takes a text as input and produces a table or several tables to summarize the content of the text. formally, the input is a text denoted as @xmath0, • • • , x |x| . the output is one table or multiple tables. for simplicity suppose that there is only one table denoted as t . further, suppose that t has n r rows and n c columns. thus, t contains n r × n c cells, where the cell of row i and column j is a sequence of words t i,@xmath1,j,1 , ..., t i,j,|t i,j | . there are three types of table: one that has both column headers and row headers, one that only has column headers, and one that only has row headers. for example, the player table in figure 1 has both column headers ("assists", "points", etc) and row headers ("al horford", "isaiah thomas", etc). we let t 1,j , @xmath2, • • • , n c denote the column headers, let t i,1 , @xmath3, • • • , n r denote the row headers, and let t i,j , @xmath4, • • • , n r , @xmath5, • • • , n c denote the non-header cells of the table. for example, in the player table in figure 1 , t 1,2 = assists, t 2,1 = al horford, and t 2,2 = 5. the information extracted via text-to-table can be leveraged in many different applications such as document summarization and text mining. for example, in figure 1 , one can quickly obtain the key information of the text by simply looking at the tables summarized from the text. there are differences between text-to-table and traditional ie settings. as can be seen from the example in figure 1 , extraction of information is performed from the entire document. the extracted information (structured data) is in a complex form, specifically multiple types of scores of teams and players in a basketball game. furthermore, the data-driven approach is taken, and the schemas of the tables do not need to be explicitly defined. the task of text-to-table also has challenges. first, parallel data containing texts and tables is difficult to obtain. manual construction of such data is usually expensive. second, structured information may not be easily represented as tables. for example, a knowledge graph may not be easily converted into tables. third, evaluation of table extraction may not be easy, which includes multiple factors, such as header, content, and structure. ## our method we develop a method for text-to-table using the seq2seq approach and the two techniques of table constraint and table relation embeddings. ## conclusion we propose employing text-to-table as a new way of information extraction (ie), which extracts information of interest from the input text and summarizes the extracted information in tables. the advantage of the approach is that one can easily conduct information extraction from either short texts or long texts to create simple tables or com-plex tables without explicitly defining the schemas. text-to-table can be viewed as an inverse problem of table-to-text. we formalize text-to-table as a sequence-to-sequence problem on top of a pretrained model. we further propose an improved method using a seq2seq model and