Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints (2020)

Abstract: Text generation from a knowledge base aims to translate knowledge triples to natural language descriptions. Most existing methods ignore the faithfulness between a generated text description and the original table, leading to generated information that goes beyond the content of the table. In this paper, for the first time, we propose a novel Transformer-based generation framework to achieve the goal. The core techniques in our method to enforce faithfulness include a new table-text optimal-transport matching loss and a table-text embedding similarity loss based on the Transformer model. Furthermore, to evaluate faithfulness, we propose a new automatic metric specialized to the table-to-text generation problem. We also provide detailed analysis on each component of our model in our experiments. Automatic and human evaluations show that our framework can significantly outperform state-of-the-art by a large margin.

## introduction understanding structured knowledge, e.g., information encoded in tables, and automatically generating natural-language descriptions is an important task in the area of natural language generation. table -to-text generation helps making knowledge elements and their connections in tables easier to comprehend by human. there have been a number of practical application scenarios in this field, for example, weather report generation, nba news generation, biography generation and medical-record description generation @xcite @xcite . most existing methods for table-to-text generation are based on an encoder-decoder framework @xcite , figure 1 : an example of table-to-text generation. this generation is unfaithful because there exists information in table not covered by generated text (marked in blue); at the same time, hallucinated information in text does not appear in table (marked in red). in this paper, we aim to overcome the above problems to automatically generate faithful texts from tables. in other words, we aim to produce the writing that a human without any external knowledge would do given the same table data as input. in contrast to existing rnn-based models, we leverage the powerful attention-based transformer model to capture long-term dependencies and generate more informative paragraphlevel texts. to generate descriptions faithful to tables, two content-matching constraints are proposed. the first one is a latent-representationlevel matching constraint encouraging the latent semantics of the whole text to be consistent with that of the whole table. the second one is an explicit entity-level matching scheme, which utilizes optimal-transport (ot) techniques to constrain key words of a table and the corresponding text to be as identical as possible. to evaluate the faithfulness, we also propose a new parent-t metric evaluating the content matching between texts and tables, based on the recently proposed parent @xcite metric. we train and evaluate our model on a large-scale knowledge base dataset @xcite . automatic and human evaluations both show that our method achieve the state-of-the-art performance, and can generates paragraph-level descriptions much more informative and faithful to input tables. ## the proposed method the task of text generation for a knowledge base is to take the structured table, @xmath0, v 1 ), (t 2 , v 2 ), , (t m , v m )}, as input, and outputs a natural-language description consisting of a -to-text generation. to enhance the ability of generating multi-sentence faithful texts, our loss consists of three parts, including a maximum-likelihood loss (green), a latent matching disagreement loss (orange), and an optimal-transport loss (blue). our model adopts the powerful transformer model @xcite to translate a table to a text sequence. specifically, the transformer is a seq2seq model, consisting of an encoder and a decoder. our proposed encoder-to-decoder transformer model learns to estimate the conditional probability of a text sequence from a source table input in an autoregressive way: ## experiments we conduct experiments to verify the effectiveness and superiority of our proposed approach against related methods. ## related work table-to-text generation has been widely studied, and seq2seq models have achieved promising performance. @xcite @xcite @xcite . for transformer-based methods, the seq2seq transformer is used by @xcite for table-to-text generation in lowresource scenario. thus, instead of encoding an entire table as in our approach, only the predicted key facts are encoded in @xcite . extended transformer has been applied to game summary @xcite and e2e nlg tasks @xcite . however, their goals focus on matching the reference text instead of being faithful to the input. another line of work attempts to use external knowledge to improve the quality of generated text @xcite . these methods allow generation from an expanded external knowledge base that may contain information not relevant to the input table. comparatively, our setting requires the generated text to be faithful to the input table. @xcite further study fidelitydata-to-text generation, where several executable symbolic operations are applied to guide text generation. both models do not consider the matching between the input and generated output. regarding datasets, most previous methods are trained and evaluated on much simpler datasets like wikibio @xcite ) that contains only one sentence as a reference description. instead, we focus on the more complicated structured knowledge base dataset @xcite that aims to generate multi-sentence texts. @xcite propose a model based on the pointer network that can copy facts directly from the input knowledge base. our model uses a similar strategy but obtains much better performance. in terms of faithfulness, one related parallel work is @xcite . however, our method is completely different from theirs. specifically, @xcite develop a confidence oriented decoder that assigns a confidence score to each target position to reduce the unfaithful information in the generated text. comparatively, our method enforces faithfulness by including the proposed table-text optimal-transport matching loss and table-text embedding similarity loss. moreover, the faithfulness of @xcite only requires generated texts to be supported by either a table or the reference; whereas ours constrains generated texts to be faithful only to the table. other related works are @xcite . for (perez-beltrachini and lapata, 2018), the content selection mechanism training with multi-task learning and reinforcement learning is proposed. for @xcite , they propose force attention and reinforcement learning based method. their learning methods are completely different from our method that simultaneously incorporates optimaltransport matching loss and embedding similarity loss. moreover, the reinforce algorithm @xcite and policy gradient method used in @xcite exhibits high variance when training the model. finally, the content-matching constraints between text and table is inspired by ideas in machine translation @xcite and seq2seq models @xcite . ## conclusion in this paper, we propose a novel transformerbased table-to-text generation framework to address the faithful text-generation problem. to enforce faithful generation, we propose a new tabletext optimal-transport matching loss and a tabletext embedding similarity loss. to evaluate the faithfulness of the generated texts, we further propose a new automatic evaluation metric specialized to the table-to-text generation problem. extensive experiments are conducted to verify the proposed method. both automatic and human evaluations show that our framework can significantly outperform the state-of-the-art methods. a parent-t metric parent-t evaluates each instance (t i , g i ) separately, by computing the precision and recall of generated text g i against table t i . in other words, parent-t is a table-focused version of parent @xcite . when computing precision, we want to check what fraction of the n-grams in g i n are correct. we consider an n-gram g to be correct if it has a high probability of being entailed by the table. we use the word overlap model for entailment probability w(g). the precision score e p for one instance is computed as follows: for recall, we only compute it against table to ensure that texts that mention more information from the table get higher scores. e r (t i ) is computed in the same way as in @xcite : thus, the parent-t score (i.e. f score) for one instance is: the system-level parent-t score for a model m is the average of instance-level parent-t scores across the evaluation set.