Improving Encoder by Auxiliary Supervision Tasks for Table-to-Text Generation (2021)

Abstract: Table-to-text generation aims at automatically generating natural text to help people conveniently obtain salient information in tables. Although neural models for table-to-text have achieved remarkable progress, some problems are still overlooked. Previous methods cannot deduce the factual results from the entity’s (player or team) performance and the relations between entities. To solve this issue, we first build an entity graph from the input tables and introduce a reasoning module to perform reasoning on the graph. Moreover, there are different relations (e.g., the numeric size relation and the importance relation) between records in different dimensions. And these relations may contribute to the data-to-text generation. However, it is hard for a vanilla encoder to capture these. Consequently, we propose to utilize two auxiliary tasks, Number Ranking (NR) and Importance Ranking (IR), to supervise the encoder to capture the different relations. Experimental results on ROTOWIRE and RW-FG show that our method not only has a good generalization but also outperforms previous methods on several metrics: BLEU, Content Selection, Content Ordering.

## introduction table-to-text generation is an essential task for text generation from structured data. it aims at automatically producing descriptive natural language text to help people obtain the salient information from the tables. over the past several years, neural text generation methods have made significant progress on this task. @xcite ; @xcite ; @xcite view the input table as a record sequence and model it as a machine translation task. to generate text containing more salient and well-organized facts, @xcite ; @xcite ; trisedya et al. (2020); name min pts ast reb … marcus smart 34 12 10 6 … amir johnson 21 2 1 3 … kelly olynyk 30 19 3 7 … … … … … … … isaiah thomas 28 29 4 3 … name pts ast reb tov … celtics 115 23 53 15 … knicks 87 19 57 25 … (a) (b) (c) name min pts ast reb … carmelo anthony 12 12 1 2 … kristaps porzingis 23 14 1 6 … joakim noah 22 9 2 10 … … … … … … … kyle o'quinn 3 2 0 3 … figure 1: (a) are tables in rotowire. (b) is a humanwritten summary related to (a). factual results that need be reasoned are in red. (c) is the entity graph constructing from the input tables. bai et al. (2020) explicitly model content selection and planning. to better represent tables, liu et al. (2018); nema et al. (2018); gong et al. (2019) explicitly model the structure of a table from multiple levels or different dimensions. figure 1 (a ) contains basketball game statistical tables from rotowire @xcite , a benchmark of nba basketball games. as can be seen, each entity (player or team) takes one row in the corresponding table. moreover, each row comprises several records of different types, which describe the entity's performance in different aspects. in terms of generating a summary from these tables, it is necessary to make reasoning to obtain some factual results from the entities' performance and the relationships between entities. for instance, when humans describe the tables in figure 1 (a), they usually give some factual results, such as "the boston celtics dominated the visiting new york knicks" or "isaiah thomas was huge for boston...". these results need to be reasoned from the entities' performance and the relationships between entities. therefore, it is necessary to give the model the reasoning ability. however, previous methods do not explicitly model this ability. numerical tables mean most records in these tables are numerical and are very common. for instance, 86.82% of the records and almost 86.49% of the column types are numeric in rotowire. we observe that there are different relations between records in different dimensions. for example, there are two kinds of relations in numerical tables. the first one is numerical size relation in the column dimension, i.e., in the same type column. the other is the relative importance relation in the row dimension. it refers to the relative importance of different types of records, which are in the same row, to the entity that they belong to. on the one hand, these relations may contribute to tableto-text generation. let us take figure 1 (a) as an example. i.thomas's score is 29, which is higher than other records in the column pts. and he has three rebounds, which is lower than most other records in the column reb. therefore, humans are more likely to describe his scores rather than his rebounds when summarizing his performance. on the other hand, a vanilla encoder may not effectively capture the relations existing in different dimensions without any auxiliary supervision. we employ a hierarchical encoder, which comprises a record encoder and a reasoning module, to encode the input tables from record level and row level. specifically, inspired by @xcite , the record encoder utilizes two cascaded self-attention modules to encode the table from the column and the row dimension, respectively. moreover, to endow the model with the reasoning ability, we first build an entity graph on the row level according to the relations between players and teams. and then, we introduce a reasoning module to perform reasoning on the graph. furthermore, we utilize different auxiliary tasks to help the encoder capture the different relations among records. more specifically, two auxiliary tasks named number ranking (nr) and importance ranking (ir) are proposed to supervise the learning of the different parts of the record encoder, respectively. we conducted experiments on rotowire and rw-fg @xcite to verify the effectiveness of the proposed approach. the experimental results demonstrate that it is necessary to enable the model the reasoning ability. moreover, the proposed two auxiliary tasks can improve the data-to-text model's performance without introducing extra parameters. furthermore, the results also show our method not only has a good generalization but also outperforms previous methods on bleu, content selection, and content ordering metrics. ## related work recently, neural models have been the mainstream for table-to-text generation and obtained impressive results. early works on table-to-text generation regard it as a distinct machine translation task and view a structured table as a record sequence @xcite @xcite . most recent works are inspired by the traditional methods for data-to-text generation and introduce explicit content selection and planning to improve the results @xcite @xcite @xcite , and they obtain training labels by aligning the input tables with related summaries. however, this alignment may introduce additional errors. some works attempt to use additional knowledge to improve the quality of the generated text. nie et al. (2018) utilize pre-executed symbolic operations on the input table in a sequence-to-sequence model to improve the fidelity of neural table-to-text generation. chen et al. (2019) introduce the background knowledge of the entity in the table to improve results. in addition to introducing external knowledge, some works learn better representation for the table by explicitly modeling the table's structure. @xcite propose a structure-aware seq2seq architecture, which incorporates the filed information as the additional inputs to the table encoder. some works @xcite @xcite model the table's representation from the row and column levels, and utilize the dual attention decoder to generate text. @xcite introduce the historical data for each table and utilize a self-attention-based hierarchical encoder on three dimensions (row, column, and time) to enrich the table's representation. furthermore, @xcite propose three auxiliary supervision tasks (sequence labeling, text auto-encoding, and multilabel classification) to help the encoder capture a more accurate semantic representation of the tables. @xcite also explicitly model the relations between the numeric records. they pretrain a multi-layer transformer encoder to obtain records' contextual numerical value representations. moreover, when training the data-to-text model, they replace the record's token embedding with its con- ## conclusion in this work, we mainly make two contributions. the first one is we introduce a reasoning module into a hierarchical table encoder, which enables the model reasoning ability. moreover, we present to utilize the different auxiliary supervision tasks to help the encoder capture the different relations between records. in detail, the number ranking (nr) task is proposed to supervise the column-wise encoder to model the numeric size relation between records in the same column. and the importance ranking (ir) task helps the row-wise encoder capture the relative importance between records in the same row. experimental results conducted on rotowire and rw-fg datasets demonstrate the effectiveness of our method. furthermore, we migrate our method to the ncp model and significantly improve its performance on rowtowire. this indicates that our proposed method has a good generalization.