Logical Natural Language Generation from Open-Domain Tables (2020)

Abstract: Neural natural language generation (NLG) models have recently shown remarkable progress in fluency and coherence. However, existing studies on neural NLG are primarily focused on surface-level realizations with limited emphasis on logical inference, an important aspect of human thinking and language. In this paper, we suggest a new NLG task where a model is tasked with generating natural language statements that can be logically entailed by the facts in an open-domain semi-structured table. To facilitate the study of the proposed logical NLG problem, we use the existing TabFact dataset~(CITATION) featured with a wide range of logical/symbolic inferences as our testbed, and propose new automatic metrics to evaluate the fidelity of generation models w.r.t. logical inference. The new task poses challenges to the existing monotonic generation frameworks due to the mismatch between sequence order and logical order. In our experiments, we comprehensively survey different generation architectures (LSTM, Transformer, Pre-Trained LM) trained with different algorithms (RL, Adversarial Training, Coarse-to-Fine) on the dataset and made following observations: 1) Pre-Trained LM can significantly boost both the fluency and logical fidelity metrics, 2) RL and Adversarial Training are trading fluency for fidelity, 3) Coarse-to-Fine generation can help partially alleviate the fidelity issue while maintaining high language fluency. The code and data are available at https://github.com/wenhuchen/LogicNLG .

## introduction neural network models, especially the recent wave of massive models like bert @xcite and gpt-2 @xcite , have shown the ability to generate natural language text at an astonishing level of fluency and coherence. for the generated text to fulfill its purpose, however, a crit- nation gold medal silver medal bronze medal sports canada 3 1 2 ice hockey mexico 2 3 1 baseball colombia 1 3 0 roller skating sentence: canada obtained 1 more gold medal than mexico. sentence: canada obtained the most gold medals in the game. medal table from tournament sentence: canada has got 3 gold medals in the tournament. sentence: mexico got 3 silver medals and 1 bronze medal. surface-level generation logical natural language generation figure 1: table-to-text generation examples with and without implicit logical inference. logical nlg requires a generation model to generate natural language statements that can be logically entailed by the facts in the table instead of simply restating certain superficial facts in natural language. however, humans have the ability to generalize beyond superficial facts (e.g., "canada has got 3 gold medals.") by inferring and communicating with new statements that can be entailed from these facts (e.g., "canada obtained the most gold medals."). we believe it is important for nlg models to be able to generalize beyond the superficla facts given to them as well. therefore, we propose a new task, logical nlg, where a model is tasked colombia has 4 medals in total. figure 2: when making the decision at the third step, the model needs to foresee the future tokens to ensure logical consistency. there is no back-tracking once the model makes a wrong decision like "5". with generating natural language statements that can be logically entailed by the given data (i.e., the premises). the new task requires a model to jointly reason and generate sentences that are consistent both linguistically and logically. since there are a variety of reasoning/inference tasks such as natural language inference @xcite and commonsense reasoning @xcite , to avoid confusion, this paper is specifically focused on inferences involving symbolic operations over the given table @xcite . to empower research in this direction, we collect a new corpus logicnlg based on the existing tabfact @xcite , which brings two major renovations to the existing nlg paradigm: 1) the text involves diversified types of logical inferences including math operations like max/min/sum/add, comparison operations like same/different, and counting operations like total/only. a more detailed description of logical inference is listed in the appendix. 2) while existing datasets are often restricted to a specific domain such as weather @xcite , restaurant @xcite , nba @xcite , etc, logicnlg uses open-domain tables without prior knowledge about their schema. as such, existing methods based on surface-level copying @xcite @xcite becomes insufficient, so are the existing fidelity evaluation based on the surfacelevel information extraction @xcite @xcite , which extracts surface triples in a certain pre-defined form (i.e. subj-pred-obj, n-gram) and compare them with the surface content given in the knowledge. most neural generation models follow a monotonic generation schema from left to right with the current prediction only depending on the preceding words. logical nlg poses unique challenges to the traditional generation scheme due to the mismatch between sequence order and logical order. as illustrated in figure 2 , the word "2" is derived from the logical inference of 'diff(silver medal of colombia, silver medal of canada)) → 2.' in other words, the logical order of word "2" should be after "more", "silver", and "canada", while the sequence order of "2" is before those words. since the monotonic generation scheme is purely based on sequence order while agnostic to logical order, existing nlg models struggle to maintain the fidelity as they cannot model the logical dependency on future tokens. to alleviate such an order mismatch, an nlg model must have the capability to plan ahead for the next few steps before generation. in this context, we believe logicnlg to be an important testbed to study such a planing/inference ability in generation models @xcite . in this paper, we further propose a non-monotonic coarse-to-fine generation model and show that it is able to alleviate the order mismatch problem and achieve better performance. the contribution of this work is three-fold: ## dataset and problem definition existing nlg datasets @xcite @xcite are mainly composed of surface-level description over the given records. though ro-towire @xcite involves sporadic inference in the long document, and the inference is restricted to domain-specific knowledge (e.g. double-double, smash, triple-double and other nba-related terms). hence, we need a better testbed for studying the proposed problem. ## automatic evaluation in this section, we discuss the evaluation of our proposed nlg task. the fluency evaluation is simply based on the standard metrics like perplexity @xcite and bleu-1,2,3 @xcite based on nltk @xcite . the most challenging problem is to evaluate the logical fidelity of the generated sentences, which is also the core problem of our paper. the existing ie-based extractive evaluation @xcite leads to two issues as shown in figure 3 : 1) empty extraction: the sentence can not be formulated as (subject, predicate, object) structure, thus the ie system fail to extract triples for verification. 2) false negative: the sentence is a logical composition (instead of surface form) of the fact from the table, the ie system cannot match it against the table. for these reasons, we test two approximate automatic metrics: ## baselines in this section, we design comprehensive baseline models to perform logical nlg. specifically, we consider the following two cases: non-pretrained models (lstm/transformer) with copy mechanism and pre-trained models (gpt-2 and bert) with sub-word unit. we train these models with three different algorithms: maximum likelihood, adversarial training, and reinforcement learning. ## coarse-to-fine generation as discussed before, the baseline models follow the monotonic generation scheme and suffer from the mismatch between sequence order and logical order (figure 2 ). in this section, we propose an imperfect remedy for such a situation based on the coarse-to-fine generation paradigm. before plunging into technical details, it is helpful to first realize the resemblance between logical nlg and semantic parsing @xcite . compared to traditional nlg tasks like machine translation and summarization, logical nlg is closer to semantic parsing in the sense that a model may make catastrophic errors that are impossible to be corrected at later steps (figure 2 ). therefore, we take inspiration from semantic parsing models @xcite that have proven effective in mitigating such errors and propose a coarse-to-fine generation scheme. we break down generation into two phases. in the first phase, 𝑃 ! [ent] gpt-2 canada obtained 1 more gold medal than mexico. obtained [ent]. more [ent] than [ent] the model only generates a template which determines the global logical structure, while in the second phase the model generates the final, grounded sentence conditioned on the template generated in the first phase. as depicted in figure 7 , we use the entity linker (section 3) to identify the entities and numbers in the original sentence y and replace them with placeholder "[ent]", which we call as the template y t . during the generation of gpt-tabgen, instead of directly predicting the final sentence y , we first predict the template y t and then y . the process is simply realized by maximizing the overall likelihood of p( ỹ |t; β), where unlike template-based or delexicalized generation @xcite , which uses rigid slot filling prone to grammatic errors, our fine-grained generation has the flexibility to modify the surface form of non-slot words, which alleviates the linguistic coherence problem @xcite . by decoupling sentence structure generation and entity grounding, our proposed coarse-to-fine scheme could partially alleviate the mismatch problem. for example, the generation of "canada" is now aware of "more than" in the latter part of the sentence, which exposes the model to more context than standard monotonic models to help make logically consistent decisions though the dependency on the "1" and "mexico" is still not captured. the proposed two-step generation could be viewed as the first step towards a fully non-monotonic generation model to solve such mismatch problem. ## experiments in this section, we explain the experimental details and then comprehensively report the automatic evaluation of different generation models and training algorithms. finally, we will conduct detailed human evaluation and error analysis. ## related work natural language generation natural language generation is a long-standing problem @xcite @xcite , which involves generating text from records or data. recently, many neural-based generation models have been proposed @xcite @xcite to achieve impressive performance on the existing datasets @xcite @xcite @xcite since the annotated text are mostly surface-level annotation without logical inference. unlike them, logicnlg has rich inference, which poses great challenges to existing models and evaluations. non-monotonic generation there have been attempts recently to study the problem of nonmonotonic text generation, which aims to teach the generation model to learn the generation order without external supervision @xcite @xcite . these models have shown to learn rational generation order to approach similar performance as the left-to-right case. these approaches are useful at capturing more sophisticated dependency within the sentence, which provides a plausible direction to pursue in logicnlg. factualness evaluation fidelity is an important research topic in generation, in rotowire @xcite and mscoco @xcite , ie-based extractive evaluation @xcite are adopted for surfacelevel matching to replace costly human evaluation. in abstractive summarization, @xcite proposes ner + relation classification method to investigate fidelity in generated summarization while @xcite proposes to use nli models to understand the entailment between generated text with the given document. these evaluations are beyond surface-level to study more sophisticated linguistic phenomena like paraphrasing, compression, entailment, inclusion, etc, which are common in summarization tasks. ## conclusion in this paper, we propose logical nlg to study the logical inference problem in generation. we conduct comprehensive experiments to show the existing nlg models are restricted by its monotonic nature and conclude this to be a proper nextstep problem to study nlg systems. there are still some unsolved problems for logical nlg, e.g. how to improve the quality of automatic metrics to better help human automatically judge models' performances. to promote the research in this direction, we host a logicnlg challenge foot_1 to help better benchmark the current progress.