L ogic2 T ext: High-Fidelity Natural Language Generation from Logical Forms (2020)

Abstract: Previous studies on Natural Language Generation (NLG) from structured data have primarily focused on surface-level descriptions of record sequences. However, for complex structured data, e.g., multi-row tables, it is often desirable for an NLG system to describe interesting facts from logical inferences across records. If only provided with the table, it is hard for existing models to produce controllable and high-fidelity logical generations. In this work, we formulate high-fidelity NLG as generation from logical forms in order to obtain controllable and faithful generations. We present a new large-scale dataset, Logic2Text, with 10,753 descriptions involving common logic types paired with the underlying logical forms. The logical forms show diversified graph structure of free schema, which pose great challenges on the model’s ability to understand the semantics. We experiment on (1) Fully-supervised training with the full datasets, and (2) Few-shot setting, provided with hundreds of paired examples; We compare several popular generation models and analyze their performances. We hope our dataset can encourage research towards building an advanced NLG system capable of natural, faithful, and human-like generation. The dataset and code is available at https://github.com/czyssrs/Logic2Text .

## introduction natural language generation (nlg) from structured data has been an important research problem in many applications. recent data-driven methods have achieved good performances on various nlg tasks @xcite @xcite . however most studies focus on surface descriptions of simple record sequences, for example, attribute-value pairs of fixed or very limited schema, like e2e @xcite and wikibio @xcite . in real-world cases for multi-row tables, it is often more desirable and plausible to provide descriptions involving higherlevel logical inference across data records. for example, in figure 1 , instead of plain restatements, human readers would be more favorable to abstract descriptions that can summarize or conclude information over the table records. to produce such logical-level generations of high fidelity, it is not yet appropriate to provide only the table as the input in a real-world nlg system, based on the following reasons: to combat with the above problems, we argue that it is necessary to leverage intermediate meaning representations to achieve faithful and controllable logical generations. to this end, we formulate the task of logical-level nlg as a logical form to text problem. specifically, besides the table information, the generation module is provided with a logical form representing the semantics of the target text (see figure 1 for an example). by separating logical reasoning and language realization, the correctness of the intermediate logical form is guaranteed, and the challenge for the realization module is fully shifted to semantic understanding. to facilitate research in this direction, we propose a new dataset named logic2text, consisting of 5.6k open-domain tables, 10.8k manually annotated (logical form, description) pairs. our dataset is of high quality in terms of (1) natural and interesting descriptions; (2) accurate logical forms with 100% execution correctness. in our dataset, the coarse logic types are 7 common ones to describe multi-row tables: count, superlative, comparative, aggregation, majority, unique, and ordinal. we employ a pythonlike program to serve as our logical forms, which can be easily converted to other types of logical forms. figure 1 shows two examples of our dataset. compared with previous surface-level nlg datasets, one major distinction of our dataset is the free schema of the logical forms, which can be represented as diversified graph structures. the new dataset poses great challenges on the model's ability to understand the structural semantics in graph representation. we employ an array of popular generation models as the baseline approaches. the experiments are conducted in (1) fully-supervised setting. we train the models using the full dataset to analyze their performances. (2) few-shot setting. we simulate the low-resource scenario in real-world use cases. experimental results show that the logical forms are critical to acquiring high-fidelity generations. the pre-trained language model outperforms other baselines @xcite , but still makes factual and logical errors. in summary, our contributions are the following: our dataset can also be used in the reverse way (text to logical form) to facilitate tasks related to semantic parsing. @xcite propose the task of fact verification against tables, however the performance is greatly limited due to the lack of logical-level nlg with logical forms ( our dataset ) logical form: eq { count { filter_eq { all_rows ; region ; africa } } ; 4 } = true description: in 2012 in opec, there were 4 member countries from africa. logical form: and { eq { hop { argmax { all_rows ; joined opec } ; region } ; africa } ; eq { hop { argmax { all_rows ; joined opec } ; country } ; angola } } = true description: in 2012 in opec, angola, from africa, was the latest country to join. the ground truth logical forms. this can be one direct application of our dataset. in this work, we focus on nlg. ## related work nlg from structured data or knowledge has been studied for many years. there are various applications, such as the automatic generations of weather reports @xcite , sport reports @xcite , clinical and health reports @xcite , response generation in task-oriented dialogue systems @xcite @xcite , etc. traditional methods typically employ a pipelinebased approach including content selection, planning and surface realization @xcite . recent data-driven methods tend to conflate the pipeline modules into one end-to-end neural networks, such as @xcite @xcite . most recently, large-scale pre-trained models @xcite @xcite have achieved new state-ofthe-arts on various generation tasks. @xcite demonstrate that a simple pre-training based method can achieve very reasonable performance on the wikibio dataset @xcite under few-shot setting. more recent works begin to focus on fidelity preserving of the generation, such as @xcite . their work obtains good performances on surface-level nlg. in contrast, our work focus on the fidelity of logical-level generations. there are a few popular nlg datasets mostly on surface-level generation. such as weath-ergov @xcite , e2e @xcite , wikibio @xcite , and totto @xcite . rotowire @xcite is a more challenging dataset on generating basketball game reports from multi-row tables. but the reports are still limited to superficial restatements of table records, with very few involving logical inference. @xcite investigate generation of interesting trivia from superlative wikipedia tables. @xcite propose the task of generating arbitrary sentences with logical inference from the table. their task mainly works for probing purpose, i.e., to test the ability of neural models to produce any logically correct descriptions solely based on the table. however, such a task formulation is not yet appropriate for building a real-world nlg system due to low-fidelity, as we discussed in the introduction. the best-performing model in @xcite only obtains a factual correctness rate over 20% based on human evaluation, which is clearly far from an acceptable level in real-world systems. another line of works related to ours is the text generation from syntactic or semantic sentence structure, such as generation from ccg grammar @xcite , ucg grammar (gardent and plainfossé, 1990), amr @xcite . there are many early works attempting algorithmic approaches on such kinds of logical formulations @xcite @xcite , etc. later proposed datasets include the groningen meaning bank (bos, 2013), the amr bank (may, 2016), the deepbank @xcite , etc. in contrast, our work focus on the logical formulations executed on database style tables, and common symbolic operations on tables, such as count, superlative, comparison. as nowadays much of the production data is stored in table based db, we believe such a dataset should help building systems with table based data. ## dataset construction the table source of logic2text is from wik-itables foot_0 @xcite , a collection of open-domain tables crawled from wikipedia. we follow @xcite to filter out overcomplicated tables and take a subset of tables with less than 20 rows and 10 columns. in this dataset, we start from 7 types of most commonly used logics @xcite to describe multi-row tables: count, superlative, comparative, aggregation, majority, unique, and ordinal. for example, for logic type count, the definition is: counting some rows in the table based on the values in one column, with the scope of all table rows or a subset. refer to appendix a for the definitions of all logic types. each description involves exactly one type of logic. this matches the observation that humans generally do not describe their interested information in tables with over-complicated logics. for logical forms, we use a python-like program, and the function set is an extension of @xcite . refer to appendix b for definitions of all functions. our dataset is constructed in 3 stages: §3.1 description composition and verification, §3.2 logical form annotation and derivation, §3.3 logical form execution and verification. we adopt the workflow of composing descriptions first and then deriving the logical forms, because under such an order, the annotators can compose natural descriptions based on the interesting facts in the table, which is hard to be achieved by automatic enumeration of logical forms followed by template re-writing. for all crowd-sourcing tasks we hire amazon mechanical turkers foot_1 (amt) under three requirements: (1) from english native countries ("us","ca","gb", "au"); ( 2 ) approval rate higher than 95% for all hits; (3) more than 500 approved hits. we follow the human subject 1980 denver broncos season date opponent game site attendance sep 7 philadelphia eagles veteran 's stadium 70307 sep 14 dallas cowboys mile high stadium 74919 sep 21 san diego chargers mile high stadium 74970 sep 29 new england patriots schaefer stadium 60153 oct 5 cleveland browns municipal stadium 81065 ... ... ... ... superlative ordinal count majority aggregation unique comparative select logic type logic type: superlative description: in the 1980 denver broncos season the highest attendance at the mile high satdium was 74970 on september 21st. ## dataset statistics and analysis we follow a rough ratio of 8:1:1 to split our dataset into 8,566 for training, 1,095 for development, and 1,092 for testing. the train, dev, and test sets have no overlap tables. we show the statistics of the dataset in ## experiments in this section we first describe the baseline models of our dataset in §5.1; then we conduct experiments in fully-supervised setting §5.2; we demonstrate the importance of the logical form in §5.3 and perform ablation studies in §5.4; at last we carry out experiments under few-shot setting §5.5.