De-Confounded Variational Encoder-Decoder for Logical Table-to-Text Generation (2021)

Abstract: Logical table-to-text generation aims to automatically generate fluent and logically faithful text from tables. The task remains challenging where deep learning models often generated linguistically fluent but logically inconsistent text. The underlying reason may be that deep learning models often capture surface-level spurious correlations rather than the causal relationships between the table \boldsymbol{x} and the sentence \boldsymbol{y} . Specifically, in the training stage, a model can get a low empirical loss without understanding \boldsymbol{x} and use spurious statistical cues instead. In this paper, we propose a de-confounded variational encoder-decoder (DCVED) based on causal intervention, learning the objective p(\boldsymbol{y}|\textrm{do}(\boldsymbol{x})) . Firstly, we propose to use variational inference to estimate the confounders in the latent space and cooperate with the causal intervention based on Pearl’s do-calculus to alleviate the spurious correlations. Secondly, to make the latent confounder meaningful, we propose a back-prediction process to predict the not-used entities but linguistically similar to the exactly selected ones. Finally, since our variational model can generate multiple candidates, we train a table-text selector to find out the best candidate sentence for the given table. An extensive set of experiments show that our model outperforms the baselines and achieves new state-of-the-art performance on two logical table-to-text datasets in terms of logical fidelity.

## introduction data-to-text generation refers to the task of generating descriptive text from non-linguistic inputs. with the different types of inputs, this task can be * corresponding authors defined more specifically, such as abstract meaning representation to text @xcite , infobox with key-value pairs to text @xcite , graph-to-text @xcite , and table-to-text @xcite generation. among these tasks, we focus on logical tableto-text generation, which aims to generate fluent and logically faithful text from tables @xcite . and the ability of logical inference is a kind of high-level intelligence, which is nontrivial for text generation systems in reality. the task remains challenging because the reference sentences often convey logically inferred information, which is not explicitly presented in the table. as a consequence, data-driven models often generated linguistically fluent but logically inconsistent text. recent progress on this task mainly lies in the use of pretrained language models (lms) like gpt-2 @xcite , which was shown to perform much better than non-pretrained models @xcite . however, it is still arguable that whether pretrained lms can correctly capture the logics, as pretrained lms like bert would use spurious statistical cues for inference @xcite . the substantial difficulty for this task does not lay on whether to use the pretrained models or not. instead, the difficulty is because the surface-level spurious correlations are easier to capture than the causal relationship between the table and the text. for example, we have observed that a model cooperating with gpt-2 generated a sentence "the album was released in the united states 2 time" for a given table. but the country where the album was released twice is "the united kingdom" 1 . in the training stage, a model may get low training loss by utilizing the surface-level correlations without 5533 actually focusing on the selected entities. as a result, in the inference stage, the model is possible to produce incorrect facts. in this paper, we view the logical table-totext generation from the perspective of causal inference and propose a de-confounded variational encoder-decoder (dcved). firstly, given the table-sentence pair (x, y), we assume confounders z c existed in the latent space and contributing to the surface-level correlations (e.g., "the united states" and "the united kingdom"). we estimate z c in the latent space based on variational inference, and cooperate the causal intervention based on pearl's do-calculus @xcite to learn the objective p(y|do(x)) instead of p(y|x). secondly, to make the latent confounder meaningful, we propose a back-prediction process to ensure the latent confounder z c can predict the not-used entities but linguistically similar to the exactly selected ones. we also consider the exactly selected entities as the mediators in our deconfounded architecture models. finally, since our variational model can generate multiple candidates, we train a table-text selector to find out the best text for the table. an extensive set of experiments show that our model achieves new stateof-the-art performance on two logical table-to-text datasets in terms of logical fidelity. the main contributions of this work can be summarized as follows: table-to-text generation. the task of table-totext generation belongs to the data-to-text generation, where a key feature is the structured input data. lebret et al. (2016) used a seq2seq neural model with a field-infusing strategy that obtains field-position-aware and field-words-aware cell embeddings to generate sentences from wikipedia tables. a follow-up work proposed to update the cell memory of the lstm by a field gate to help lstm identify the boundary between different cells @xcite . transformerbased @xcite models were also proposed which improved the ability to capture longterm dependencies between cells @xcite @xcite . it is worth to mention that the copy mechanism @xcite is an important part to deal with the outof-vocabulary (oov) words @xcite @xcite when not using pretrained language models. logical table -to-text generation. while usually fluent, existing methods often hallucinate phrases that contradict the facts in the table. to benchmark models' ability to generate logically consistent sentences, recent work proposed a dataset collected from open domain @xcite , which would score low on those models ignoring logical consistency. follow-up work further proposed another dataset that involved logical forms as additional supervision information @xcite , which includes common logic types paired with the underlying logical forms. ## backgrounds before introducing our models, we briefly review the framework of vae (kingma and welling, 2014), a generative model which allows to generate high-dimensional samples from a continuous space. in the probability model framework, the probability of data x can be computed by: when applied to seq2seq generation where the input and the output are denoted by x and y respectively, the conditional variational autoencoder (cvae), or often known as variational encoder-decoder (ved), is used with following approximation: in the vanilla cvae formulation, such as the ones adopted in @xcite , the prior distribution p(z|x) is approximated to p(z), which is independent on x and fixed to a zero-mean unit-variance gaussian distribution n (0, i). however, this formulation is shown to induce a strong model bias @xcite and empirically perform worse than non-variational models @xcite in multi-modal situation. figure 1 : the causal graphs before and after the docalculus. the symbols x, y, z m , z c denote the input table, the output sentence, the hidden mediator, and the hidden confounder, respectively. we assume c and m to be the proxy variables of z m and z c , respectively, which are relatively easy to be observed. ## conclusion in this paper, we propose a de-confounded variational encoder-decoder for the logical table-to-text generation. firstly, we assume two latent variables existed in the continuous space, representing the mediator and the confounder respectively. and we apply the causal intervention method to reduce the spurious correlations. secondly, to make the latent variables meaningful, we use the exactly selected entities to supervise the mediator and the not selected but linguistically similar entities to supervise the confounder. finally, since our model can generate multiple candidates for a table, we train a selector guided by both surface-level and logical fidelity to select the best sentence. the experiments show that our model yields competitive results with recent sota models.