How Helpful is Inverse Reinforcement Learning for Table-to-Text Generation? (2021)

Abstract: Existing approaches for the Table-to-Text task suffer from issues such as missing information, hallucination and repetition. Many approaches to this problem use Reinforcement Learning (RL), which maximizes a single manually defined reward, such as BLEU. In this work, we instead pose the Table-to-Text task as Inverse Reinforcement Learning (IRL) problem. We explore using multiple interpretable unsupervised reward components that are combined linearly to form a composite reward function. The composite reward function and the description generator are learned jointly. We find that IRL outperforms strong RL baselines marginally. We further study the generalization of learned IRL rewards in scenarios involving domain adaptation. Our experiments reveal significant challenges in using IRL for this task.

## introduction table -to-text generation focuses on explaining tabular data in natural language. this is increasingly relevant due to the vast amounts of tabular data created in domains including e-commerce, healthcare and industry (for example, infoboxes in wikipedia, tabular product descriptions in online shopping sites, etc.). table -to-text can make data easily accessible to non-experts and can automate certain pipelines like auto-generation of product descriptions. traditional methods approached the general problem of converting structured data to text using slot-filling techniques @xcite @xcite @xcite . while recent advances in data-to-text generation using neural networks @xcite @xcite @xcite have led to improved fluency, current systems still suffer from issues such as lack of coverage (where the generated text misses information present in the source), repetition (where the generated text repeats information) and hallucination (where the generated text asserts information not present in the source) @xcite . a significant reason for these issues is that models often lack explicit inductive biases to avoid these problems. most extant approaches utilize reinforcement learningbased (rl) training, using a single reward (such as bleu or task-specific rewards) that optimizes for a specific aspect. for example, @xcite and @xcite use domain-specific rewards to improve the accuracy of medical report generation. however, defining a single reward that addresses all of the above-described issues is difficult. to use multiple reward components with rl, one has to manually find an optimal set of weights of each component either through a trial-and-error approach or expensive grid search which gets infeasible as the number of such reward components increases. inverse reinforcement learning @xcite @xcite can be a natural approach for this task since it can learn an underlying composite reward function from labeled examples incorporating multiple rewards. motivated by existing applications of irl in other domains and tasks @xcite @xcite , we explore its utility for table -to-text generation. we diverge from previous work on irl in designing a set of intuitive and interpretable reward components that are linearly combined to get the reward function. figure 1 illustrates the overall idea of this work. we learn a "description generator" (also referred as policy later) to generate descriptions given a table. the irl framework includes "reward approximator" that leverages the "expert" or the ground-truth de- slot type row slot value name 1 asgar dob 2 27 march 1960 ... ... ... occ. 5 footballer table: update generator update approximator update approximator reward approximator recall repetition fluency bleu reconstruction generated description expert description: asgar ( born 27 march 1960 ) is a retired footballer ... description generator decoder hidden states @xmath0,5) <occ.> footballer (5,1) sos asgar ... . eos asgar encoder hidden states ... table-to-text task under inverse reinforcement learning framework using multiple reward components scriptions corresponding to tables to jointly learn the underlying composite reward function combining multiple reward components such as "recall", "fluency", etc. this composite reward function quantifies the quality of the generated descriptions. we see irl performs at par with rl baselines. for investigating when irl helps and when it does not, we conduct experiments to evaluate generalization capabilities of irl in limited data setting and identify challenges involved in irl training. our contributions are: • we formulate a set of interpretable reward components and learn the composite linear reward function in a data-driven manner for table-to-text generation 1 . • we study the utility of irl for table-to-text generation. the training data for this task consists of pairs of tables and corresponding natural language descriptions, as shown in figure 1 . a table t is a sequence of tuples of slot types (e.g. "name") and slot values (e.g. "asgar") and let d denote the expert description. we formulate the "table -to-text" task as generating d from source table t . in the rest of this section, we first explain how to formulate table -to-text under the irl framework, followed by the formulation of the reward components and a brief description of the text generation network 1 code and dataset splits for the paper are provided in https://github.com/issacqzh/irl_ table2text that is at the core of our method. ## experiments and results in this section we describe our experiments and their results in detail. @xcite proposed a dataset of tables and their corresponding descriptions related to people and animals from wikipedia. however, the original released dataset is noisy (many descriptions have low precision/recall, most examples have very few distinct slot types, etc.). for our experiments, we filtered this dataset to get a smaller high-quality dataset of 4623 examples using the following criteria : (1) recall (defined in §2.2) of 1.0 (2) high precision (fraction of entities in the description mentioned in the table) greater than 0.7 (3) number of distinct slot types greater than 6. we split the entire dataset as 80%, 10% and 10% for training, validation and testing respectively. details of the dataset are provided in appendix a.2. to aid reproducibility we make the data splits used by us publicly available foot_0 . ## discussion we highlight some challenges with irl training that potentially hinder irl to get significantly better than rl baselines. further, we discuss qualitative differences between rl and irl models. ## conclusion we present an approach using irl for table -to-text generation using a set of interpretable reward components. while the approach outperforms rl, improvements are marginal, and we identify several challenges. in particular, using metrics like bleu as reward components is problematic, since they affect weight convergence for irl. based on our study, the application of irl for table -to-text generation would broadly benefit from designing better-calibrated reward components and improvements in training paradigms. we hope our exploration encourages the community to engage in interesting directions of future work. the gradient w.r.t. reward parameters is given by the partition function requires enumerating all possible descriptions which makes this intractable. this is tackled by approximating the partition function by sampling descriptions from the policy using importance sampling. the importance weight β i for a generated description d i is given by the gradient is now approximated as: