Towards Table-to-Text Generation with Numerical Reasoning (2021)

Abstract: Recent neural text generation models have shown significant improvement in generating descriptive text from structured data such as table formats. One of the remaining important challenges is generating more analytical descriptions that can be inferred from facts in a data source. The use of a template-based generator and a pointer-generator is among the potential alternatives for table-to-text generators. In this paper, we propose a framework consisting of a pre-trained model and a copy mechanism. The pre-trained models are fine-tuned to produce fluent text that is enriched with numerical reasoning. However, it still lacks fidelity to the table contents. The copy mechanism is incorporated in the fine-tuning step by using general placeholders to avoid producing hallucinated phrases that are not supported by a table while preserving high fluency. In summary, our contributions are (1) a new dataset for numerical table-to-text generation using pairs of a table and a paragraph of a table description with richer inference from scientific papers, and (2) a table-to-text generation framework enriched with numerical reasoning.

## introduction recent data-to-text generation studies have shown significant improvement in generating faithful text aligned with data sources. a copy mechanism has been widely explored to improve faithfulness in various ways. @xcite used joint probabilities to let models choose between copying records from data sources or generating from a vocabulary. @xcite improved a similar approach by modeling entity representations as a unit of copying. this approach has proven to be effective in generating descriptive text that explicitly mentions facts from sources. however, as introduced by @xcite , humans have the ability to produce more analyti the power of tables in presenting data efficiently further encourages research done by exploring the tables as data sources in natural language tasks, such as table-to-text generation @xcite @xcite , table question answering @xcite , and table-based fact verification @xcite . recent research on the ## numerical table-to-text dataset we created numericnlg, a new table-to-text dataset focusing on a text generation task with numerical reasoning. we collected table descriptions from scientific papers, that are naturally produced by experts with richer inference. ## table representation due to rotowire's limited schemas, @xcite viewed a table input as a set of records (entity, value, type), where the entity and the type are the extracted row and column names, respectively. because of the unlimited table schemas in our dataset, by capturing the original table structure in real-world tables, this paper uses the representations which consist of captions, row headers, column headers, cell values, and metrics, called a data table . using only descriptive facts from the data table as input representations is sufficient to generate descriptive texts that explicitly mention facts in the table. however, since we intend to produce more analytical text with numerical reasoning, we propose adding inferred facts to the input representation by computing a set of arithmetic operations on the data table beforehand, defined as a pre-executed operation table. data table we view t as a set of cells with their corresponding row header (rh), column header (ch), numerical value (val), and metric-type (m), defined as a data table (t d ). a data table for the example in figure 1 consists of rh: ((model, our full model), @xcite )); ch: (); val: ((89.6, 82.2, 85.7), (86.2, 83.7, 84.9); and m: (precision, recall, f1). since our tables are annotated with a targeted header as a content plan for table descriptions, we mark cells corresponding to the targeted header with a target flag (tgt) to highlight the marked cells in text generation. we set @xmath0. in this study, we preprocess the header name by concatenating the row and column headers (@xmath1; ch]) and keep information about the header category by extracting overlapping tokens of row and column headers as th. as a result, we define t @xmath2, th ij , val ij , m ij , tgt ij ), where 1 ≤ @xmath3, 1 ≤ @xmath4; n r and n c are the numbers of rows and columns, respectively. pre-executed operation table we provide a table of pre-executed cell operations (t op ) by doing mathematical operations only on targeted cells to limit the calculation. in this study, we cover maximum, minimum, and difference operations. examples of a preprocessed table, data table, and pre-executed operation table are shown in figure 2 . supporting transfer learning of pre-trained transformers to our table-to-text generation task, we prepare a linearized table p t as an input representation so that it similar to the representation that encoder has seen during pre-training. t is converted to a flat string p @xmath5, ..., w |p t | , similar to that used in many prior work @xcite @xcite , where w i denotes the i-th word in paragraph p t with length |p t |. in this study, we adopt the template-based input representation, introduced by @xcite , to handle representation bias between a structured data t and a natural language utterance p t , where p t is generated using a manually defined template. we propose not only covering data table t d in the template but also injecting the pre-executed numerical operations of table t through t op to guide numerical-reasoningbased text generation. we consider four different methods foot_1 for converting t into sequences, the last two being our contributions. ## generation models the task is to generate text by translating table representation p t into table description @xmath6, y 2 , ..., y n . we apply a series of generation models to solve the proposed task. while our focus is primarily on pre-trained models since they have been most widely used for limited data settings, <@xmath7. t op of ## experiments we conducted experiments on the proposed dataset to evaluate the performance of the text generation models and verify the effectiveness of the approach of using different table representations. ## conclusion we proposed numericnlg, a new dataset for tableto-text generation using a table and its corresponding description from scientific papers, focusing on numerical-reasoning texts. even though our proposed dataset is not a large-scale table collection, we provided pairs of a table and its rich inference description, that are naturally written by experts in scientific papers, supporting further research on table -to-text generation with numerical reasoning. we conducted experiments with fine-tuned pretrained models by using several types of table linearization as input representations, comparing with a template-based generator and pointer-generator. the experiments showed that transfer-learning of pre-trained language models leads to an improvement in our settings, that resulted in more fluent text while it still lacked fidelity to table contents. we then proposed incorporating a copy mechanism by using general placeholders to avoid the production of hallucinated phrases, that are not supported by tables while preserving high fluency. even though our proposed copy mechanism failed to learn to generate better outputs in the decoder-only pre-trained models, we showed that a copy-based pre-trained model with an encoder-decoder archi-tecture leads to a better bleu score and improves correctness. model generated text fine-tuned gpt2 (naive) we compare our results using the model with that of the other model, which shows the performance on both models. we can conclude that the models are more reliable in predicting the usefulness of the model and more reliable than the other models. we suggest that the more accurate the model, the higher the mention detection results on both models. the model with the highest mentions detection yields a better model, and also shows the best performance on both models. fine-tuned gpt2 (td temp) and finally we have seen that our model does not do the right thing by simply using the word "we." as a noun we do not do the right thing by using the verb "we." as a noun our word "we" does not even have a verb, "we do." as a verb we do not even have a verb, "we do. fine-tuned gpt2 (top temp) the table shows the recall results for our full model. the f1 performance was better than that in our full model, indicating that we did not have any other models in the dataset. the rec. performance was comparable to that for the full model with a f1+ performance of 82.7%. fine-tuned gpt2 (naive) + copy