PLOG : Table-to-Logic Pretraining for Logical Table-to-Text Generation (2022)

Abstract: Logical table-to-text generation is a task that involves generating logically faithful sentences from tables, which requires models to derive logical-level facts from table records via logical inference. It raises a new challenge on the logical-level content planning of table-to-text models. However, directly learning the logical inference knowledge from table-text pairs is very difficult for neural models because of the ambiguity of natural language and the scarcity of parallel data. Hence even large-scale pre-trained language models present low logical fidelity on logical table-to-text. In this work, we propose a Pretrained Logical Form Generator (PLOG) framework to improve generation fidelity. Specifically, PLOG is first pretrained on a table-to-logical-form generation (table-to-logic) task, then finetuned on downstream table-to-text tasks. The logical forms are formally defined with unambiguous semantics. Hence we can collect a large amount of accurate logical forms from tables without human annotation. In addition, PLOG can learn logical inference from table-logic pairs much more reliably than from table-text pairs. To evaluate our model, we further collect a controlled logical table-to-text dataset CONTLOG based on an existing dataset. On two benchmarks, LOGICNLG and CONTLOG, PLOG outperforms strong baselines by a large margin on the logical fidelity, demonstrating the effectiveness of table-to-logic pretraining.

## introduction table-to-text generation is a sub-task of data-to-text generation, aiming to generate natural language descriptions from structured tables. there are two main steps to performing table-to-text generation: content planning (selecting table contents and determining the plan to describe them) and surface realization (verbalizing the plan into fluent natural language). traditional table-to-text systems adopt a pipeline architecture to complete the two procedures with separate modules @xcite . recent work has shown the advantage of using a neural encoder-decoder model to directly generate sentences from the tables, which presents the strong capability to produce fluent and natural text @xcite @xcite . researchers have also attempted to finetune pretrained language models such as bart @xcite and t5 @xcite on downstream table-to-text tasks and achieved remarkable success on a broad range of benchmarks @xcite . previous studies have mainly focused on surfacelevel realization, i.e., simply restating surface-level facts in natural language @xcite @xcite . recently, logical table-to-text generation @xcite , i.e., generating textual descriptions that require logical reasoning over surface-level facts in the table, has attracted increasing attention. logical table-to-text generation poses a new challenge of logical-level content planning, requiring models to perform logical inference to derive facts from surface-level table records. end-to-end neural models often suffer from low logical fidelity on this task, i.e., the generated sentences are not logically entailed by the tables despite their reasonable fluency @xcite . we attribute this to the fact that the ambiguity of natural language target sentences hinders neural models from learning accurate logical inference from table-text pairs. in addition, the amount of such table-text pairs is limited because of the labor-intensive human annotation for logic-focused descriptions, which also limits the performance of neural models. to achieve high fidelity of logical-level generation, @xcite have attempted to annotate branding callsign frequency power location dwbl 1242 dwbl 1242khz 20kw metro manila mellow 947 dwll 94.7mhz 20kw metro manila mellow 887 dyku 88.7mhz 10kw iloilo city mellow 957 dxbl 95.7mhz 10kw cagayan de oro mellow 947 davao dxll 94.7mhz 20kw davao city (b) (controlled) logical table-to-text generation input: table + (highlighted cells) target: for dwbl, the power for dwll is higher than the power for dyku. (a) table-to-logical-form generation input: table + (highlighted cells) target: greater { hop { filter_eq { all_rows ; callsign ; dwll } ; power } ; hop { filter_eq { all_rows ; callsign ; dyku} ; power } } title: dwbl intermediate pretraining on task (a) finetuning on task (b) pretrained lm plog plog( finetuned) figure 1: examples of the tasks and the training procedure of our proposed plog model. task (a) is the table-tologic pretraining task we propose; task (b) is the downstream logical table-to-text task we target. the yellow-colored table cells are annotated as control features for the contlog task, while for logicnlg, such highlighted cells are not available. we collect different table-to-logic datasets for contlog and logicnlg separately and perform intermediate pretraining for pretrained language models on the collected data, then finetune the model on the downstream tasks. logical forms to guide the text generation and proposed a logic2text dataset. with logical forms as mediators conveying accurate logical-level facts, models can focus on surface realization from associated logical forms and achieve high fidelity. however, annotating accurate logical forms for textual descriptions requires intensive human efforts. moreover, generating from a self-contained logical form is actually a different task from table-to-text generation. prior studies on this dataset (liu et al., 2021a; shu et al., 2021; xie et al., 2022) mostly focus on converting the logical forms into texts rather than tables into texts. in this study, we propose a pre-trained logical form generator (plog) model to achieve more faithful logical table-to-text. specifically, plog is first pretrained on a large-scale synthetic corpus of table-to-logical-form generation (table-to-logic) to learn how to generate accurate logical forms from tables, then finetuned on downstream table-to-text tasks to transfer the logical inference knowledge learned from pretraining to text generation. our insights are three-fold. (i) unlike natural language sentences, logical forms are formally defined with unambiguous semantics; hence it is much easier and more reliable for models to acquire logical inference knowledge via learning from logical form generation. (ii) it is viable to collect large-scale logical form corpora via rule-based sampling over tables without the efforts of human annotators. (iii) via pretraining on large amounts of table-to-logic data, the proposed model can better understand the table and organize the logical-level content planning, leading to faithful table-to-text generation. here, we treat logical forms as intermediate mean-ing representations of logical-level texts, while we do not need them when performing the downstream task. to collect the pretraining data, we propose an execution-guided sampling approach to sample accurate logical forms from tables automatically. we formulate the pretraining task in the same sequence-to-sequence (seq2seq) generation to achieve smooth transfer learning to the downstream table-to-text task. we adopt several strong pretrained language models, bart and t5, as the backbone models. because the previous benchmark for logical table-to-text, logicnlg, lacks control features, leading to uncontrollable content selection and poor logical fidelity, we collect a a new controlled logical natural language generation (contlog) dataset as a complementary testbed towards controlled logical table-totext generation. specifically, we re-organize the logic2text dataset by detecting highlighted cells based on their annotated logical forms. figure 1 presents examples of the table-to-logic pretraining task and the (controlled) logical table-to-text task. on the two benchmarks, logicnlg and con-tlog, plog outperforms the strong baselines such as t5 by a large margin on the logical fidelity, demonstrating the effectiveness of table-to-logic pretraining. human evaluation and analysis experiments further demonstrate that our approach can considerably promote the fidelity of logical tableto-text generation. 1 2 related work table -to-text generation early table-to-text generation tasks are limited to surface-level generation with little focus on logical inference @xcite . logicnlg is the first dataset to focus on logical table-to-text generation, with wikipedia tables and human-annotated logical descriptions. @xcite proposed a deconfounded variational encoder-decoder model to encourage the model to generate non-surface-level predictions; however, the logical reasoning process is not explicitly considered, leading to low fidelity scores on human evaluation. @xcite proposed to annotate logical forms to guide the generation and released a logic2text dataset. in this work, we focus on direct logical table-to-text generation without any explicit logical forms. another related line of datasets are totto @xcite and hitab @xcite , which incorporate highlighted cells to promote controllable generation. the contlog dataset we propose is similar to the task settings of these datasets but differs in that we focus on logical-level generation. at the same time, only a small portion of examples in totto and hitab involve logical reasoning. rotowire @xcite and @xcite also involve numerical reasoning over table records, while they focus on document-level table summarization instead of sentence generation. table pretraining table pretraining @xcite @xcite has been popular for table understanding tasks such as table question answering (tableqa) @xcite and table fact verification (tablefv) @xcite . with large-scale pretraining corpora, the table pretraining models can learn a better joint understanding of tabular and textual data through well-defined pretraining objectives. most table pretraining works are based on table-text corpora, while tapex @xcite learns from synthetic sql programs, which is the closest to our work. specifically, tapex is first pretrained on a table-based sql execution task, where the input is a table and a sql program, and the output is the answer to the sql query. then, the pretrained model can be finetuned on tableqa and tablefv tasks where the input is a table associated with a textual query/statement, and the output is the an-swer. however, our work differs from tapex in that we focus on table-to-text generation, where the input is a structured table and the output is a textual statement of the table contents. our task requires deriving a complete logical-level fact from the table without the guidance of any query. in addition, our pretraining task also requires generating a self-contained logical form from the table, while tapex aims to learn the neural execution of an existing sql program. similarly, flap (anonymous, 2021) proposes to enhance the numerical reasoning ability of table-to-text models with an artificial pretraining task. this task is a synthetic qa task similar to tapex pretraining. another line of related works adopts pretraining techniques to solve the text-to-sql parsing @xcite task, also involving collecting synthetic sql data and pretraining models on sql generation tasks. however, text-to-sql still requires an explicit nl query as the input, which is different from our task. although table pretraining is popular in table understanding tasks, it has not been well-explored in table-to-text. previous works on table-to-text tend to directly utilize pretrained language models by flattening structured tables into sequences @xcite @xcite . a recent work @xcite incorporates structural positional embeddings of tables into t5 @xcite and performs intermediate pretraining in a similar way to tapas @xcite . similarly, plog can also be seen as intermediate pretraining of language models for table-to-text generation. ## downstream tasks in this work, we focus on logical table-to-text. the previous benchmark logicnlg aims at generating sentences from a full table without control features, which causes uncontrollable content selection and hinders faithful generation @xcite . therefore, we propose a new controlled logical table-to-text dataset contlog as a complementary testbed to logicnlg. inspired by previous studies on controlled table-to-text @xcite , we incorporate highlighted cells as additional supervision signals in contlog (figure 1 ) to narrow down the scope of content selection, such that models can focus more on planning and generation. ## table-to-logic pretraining logical table-to-text is difficult mainly because of the ambiguity of natural language sentences. for example, a sentence alice was the first player that achieved champion in 2010 has two possible meanings: (1) alice got the first champion of 2010; (2) alice became the first champion in history, and this achievement happened in 2010. this prevents end-to-end neural models from inferring unambiguous logical facts from the table, especially when the parallel data are scarce. to achieve faithful logical table-to-text generation, we propose a table-to-logic pretraining task that involves generating a logical form from an input table. in this task, the model needs to mine logical-level facts from tables and organize the facts into formally defined meaning representations, i.e., logical forms. each logical form can be regarded as an abstract content plan of a logical-level description. therefore, we expect a model to learn logical-level content planning from the pretraining task. we then finetune the model on the downstream table-to-text tasks to generalize the content planning to natural language generation. we formulate our pretraining and downstream tasks as the same seq2seq generation paradigm to realize successful transfer learning. ## the plog model in this section, we introduce our proposed model plog and how we conduct the seq2seq generation for the pretraining and downstream tasks. backbone model we utilize the same backbone model to address both tasks to achieve the knowledge transfer from the table-to-logic pretraining task to the table-to-text downstream task. theoretically, any text generation model applies to our task, such as gpt-2 @xcite , bart, and t5. we test different backbone models, including bart-large, t5-base, and t5-large. ## experiments 6.1 experimental settings evaluation metrics following prior works @xcite on log-icnlg, we evaluate our models on both surface-level matching metrics and logical fidelity scores. surface-level metrics include bleu-1/2/3, which are based on n-gram matching between the model generations and gold references. in terms of fidelity scores, prior works adopt sp-acc and nli-acc. for sp-acc, a sentence is first parsed into a logical program and evaluated as the execution accuracy of the program. nli-acc is based on tablebert, a table-entailment model pretrained on the tabfact dataset @xcite . the model can predict whether a table supports a sentence. however, these two fidelity metrics are not enough to verify the fidelity: we empirically find that the parsing algorithm for sp-acc often generates irrelevant logical programs for the sentences, which renders the evaluation inaccurate. in addition, the tablebert model used for nli-acc only achieves 65.1% accuracy on the tabfact dataset, and we find it overly positive about the predictions. to this end, we add two state-of-the-art table-entailment models for evaluation: tapexlarge @xcite and tapas-large (eisenschlos et al., 2020), which achieve 84.2% and 81.0% test accuracy on tabfact, respectively. we name the two metrics as tapex-acc and tapas-acc, respectively. we still evaluate sp-acc and nli-acc to compare our method with previous studies. for contlog, we adopt the evaluation metrics of logic2text: bleu-4 and rouge-1/2/4/l to evaluate surface-level matching, and use tapex-acc and tapas-acc to evaluate the fidelity. models for comparison for logicnlg, we compare our method with the following models: gpt-tabgen (sm) and gpt-coarse-to-fine (sm) @xcite are two baselines based on pretrained gpt-2; dcved+gpt-tabgen @xcite is a de-confounded variational model with gpt-tabgen (sm) as the backbone. we also include pretrained bart-large, t5-base and t5-large as the baselines models for both logicnlg and contlog, for which we adopt our data pre-processing method introduced in section 5. our models are named plog (bart-large), plog (t5-base) and plog (t5-large) when using different backbones. we adopt the same input serialization strategy with numerical pre-computation for bart, t5, and plog models. training details we conduct our main experiments based on transformers @xcite and pytorch @xcite . during training, the parameters of embedding layers of models are frozen. during inference, we adopt beam search with beam size 4 for all the experiments. we set the maximum length as 500 and 200 for source and target sequences, respectively. each experiment was run only once because of the time cost. on logicnlg, model selection is based on the bleu-3 score on the validation set, and on cont-log, it is based on validation bleu-4 scores. the selection of pretraining checkpoints is based on the execution accuracy of generated logical forms on the validation set of pretraining tasks. we provide detailed hyperparameters in appendix a. ## conclusion we proposed a table-to-logic pretraining task to enhance the fidelity of logical table-to-text generation. in addition, we constructed a controlled logical table-to-text generation task by re-purposing an existing dataset. to realize pretraining on largescale corpora, we proposed an execution-guided sampling scheme to extract accurate logical forms from tables automatically. with table-to-logic pretraining, our table-to-text model could significantly improve logical fidelity. our work shows a novel way to utilize formal language to promote tableto-text generation, and may be extended to other related areas such as table representation learning.