Plan-then-Seam: Towards Efficient Table-to-Text Generation (2023)

Abstract: Table-to-text generation aims at automatically generating text to help people conveniently obtain salient information in tables. Recent works explicitly decompose the generation process into content planning and surface generation stages, employing two autoregressive networks for them respectively. However, they are computationally expensive due to the non-parallelizable nature of autoregressive decoding and the redundant parameters of two networks. In this paper, we propose the first totally non-autoregressive table-to-text model (Plan-then-Seam, PTS) that produces its outputs in parallel with one single network.PTS firstly writes and calibrates one plan of the content to be generated with a novel rethinking pointer predictor, and then takes the plan as the context for seaming to decode the description. These two steps share parameters and perform iteratively to capture token inter-dependency while keeping parallel decoding. Experiments on two public benchmarks show that PTS achieves 3.0 5.6 times speedup for inference time, reducing 50% parameters, while maintaining as least comparable performance against strong two-stage table-to-text competitors.

## introduction table-to-text generation @xcite is a long-standing problem that aims to generate natural language descriptions of structured table data. a good table-to-text system can help end users better identify the informative elements and their relations from a table. therefore, developing table-to-text systems is of tremendous value in a wide range of applications, such as biography generation @xcite , basketball news generation @xcite , advertising text generation @xcite , and table-based question answering @xcite . table encoder plan decoder reference: thaila ayala ( born april 14, 1986 in brazil ) is an actress and model . stage 1: content planning (b) (a) input table: @xmath0: thaila ayala april 14 1986 brazil actress model t attention [bos] … brazil actress model [eos] thaila … brazil actress model plan encoder text decoder stage 2: surface generation thaila attention [bos] … and model . [eos] thaila … and model . figure 1: (a): example of table-to-text generation from wikibio. tokens from the content planning are colored in red. (b): two-stage model, which disentangles tableto-text generation into two stages: content planning and surface generation. recently, neural network-based approaches have made significant progress in this field. the modern neural models for table-to-text generation can be roughly categorized into one-stage models and twostage models. one-stage models generate natural language descriptions directly from the table by simply relying on representation learning to generate well-organized fluent descriptions. along this line, some studies propose to modify the model architectures to effectively learn from structured table @xcite , while some other works introduce auxiliary tasks to help the encoder capture a more accurate semantic representation @xcite . the major drawback of one-stage models is the lack of interpretability and controllability, making the models prone to suffer from unfaithful hallucinations @xcite . to alleviate the aforementioned shortcoming of one-stage models, some researchers propose a twostage paradigm for table-to-text generation @xcite , which explicitly decomposes the whole generation process into two separate stages: content planning and surface generation (illustrated in figure 1 ). the content planning model generates an intermediate sequence that specifies the tokens to be verbalized. the generated plan provides some interpretability and controllability, thus can potentially reduce the risk of hallucinations @xcite . the surface generation model then completes the description based on the intermediate plan. although two-stage models have some superiority over one-stage models, they are often computationally intensive. the cause of the high computational cost is two-fold. firstly, most twostage models use autoregressive (ar) decoders, which is quite time-consuming due to their nonparallelizable nature, especially for long sequences @xcite . secondly, the two-stage systems often consist of two different models, which usually double the parameter scale (see section 4.4). the increased parameter scale may introduce more computation overhead and thus slow down the inference speed. these disadvantages limit the deployment of current neural table-to-text systems in practical applications. recently, non-autoregressive (nar) generation has attracted much attention because it can significantly accelerate inference speed for text generation @xcite @xcite . however, as demonstrated in our preliminary experiments (see also section 4.4), applying the nar models directly to table-to-text generation may suffer from lower generation quality because nar models do not explicitly model the content planning procedure, which can provide good initial input for nar decoder. in this work, we propose to reduce the computational cost of two-stage models through a unified nar framework, which is called plan-then-seam (pts). pts is a iterative nar table-to-text model. specifically, pts first generates the content plan in the first iteration. then it fills in the other surface tokens in subsequent multiple iterations to seam the intermediate plan tokens. note that pts conducts the content planning and surface generation tasks in a single model, thus the model size is smaller than previous two-stage models. moreover, since pts is a nar model, it is more efficient than the ar counterparts. given that fully nar content planning may ignore the dependency between planned tokens, we introduce a rethinking pointer predictor, which can better calibrate the planning conditioned on the generated ones. our contributions can be summarized as follows: ## related work table -to-text generation has long aroused interest in the natural language generation (nlg) community @xcite . recently, neural models have been the mainstream for this task and made impressive progress. models for this task can be mainly categorized into two types: one-stage models and two-stage models. one-stage models generate text directly from structured data through a neural encoder-decoder architecture @xcite . they simply rely on representation learning to improve the generation. @xcite propose a structure-aware seq2seq architecture, which incorporates the filed information as the additional inputs to the table encoder. some works design hierarchical table encoder which model table's representation from the row and column levels @xcite . @xcite ; @xcite introduce auxiliary supervision tasks to help the encoder capture a more accurate semantic representation of the tables. however, one-stage methods are prone to produce unfaithful hallucinations and uncontrollable generation @xcite . as the improvement, neural two-stage models @xcite @xcite @xcite decompose the table-to-text generation into content planning and surface generation stages. in general, content planning is implemented by pointer networks @xcite . the explicit content planning mechanism not only decomposes the complex table-to-text generation into two easier tasks but also makes the generation process more interpretable and controllable by generating an intermediate representation. however, the hallucination problem persists in the surface generation stage as it is autoregressive (ar). to address this issue, sana @xcite proposes an edit-based non-autoregressive (nar) surface generation model that generates texts through iterative insertion and deletion operations while maintaining an ar planning stage. existing twostage methods solely pay attention to improving the generation quality while ignoring its efficiency. compared with one-stage models, two-stage methods double the number of parameters. additionally, the ar generation is slow at inference time. these problems hinder the practical deployment of current neural table-to-text models. ## methodology given a region of a table as input, table-to-text generation is to produce a natural language description @xmath1, ..., y n } to describe the selected table region. this paper proposes the first totally nonautoregressive table-to-text model, plan-then-seam (pts). as depicted in figure 2 , pts consists of three major components, a table encoder, a nonautoregressive content planning decoder (nar-p), and a non-autoregressive seaming decoder (nar-s), which collaborate to generate a description for a source table in an iterative manner. at the first iteration, nar-p generates content planning sequence in a fully non-autoregressive manner by conditioning on the source table. at the subsequent iterations, nar-s seams the content planning tokens by inserting connective tokens between them to generate a fluent description. next, we will describe the proposed pts in detail. ## human evaluation to verify whether the system performance is consistent with what the automatic metrics show, we further conduct a human evaluation on the wikibio test set. we randomly sample 50 instances from each model's generated outputs. then, we invite three graduate students, whose english level is very high to understand the text, to score each generated text from 1 to 5 in terms of two criteria: fluency (is the sentence fluency?) and faithfulness (is the sentence related to the input table ?). for each criterion, we average the scores from all annotators as the final score. when evaluating, each annotator is provided the input tables as the references and does not know which model the generated text comes from. the results are summarized in ## conclusion we propose a unified non-autoregressive framework, plan-then-seam (pts), for table-to-text generation. given a source table, pts first generates the content plan in a fully nar manner. then we iteratively fill in the other surface tokens. experimental results demonstrate that pts achieves a 3.0 ∼ 5.6 speedup with only 50% model parameters compared with previous two-stage table-to-text models, without degrading the description quality. further analysis reveals that the success of pts comes from the proposed nar-p with a rethinking mechanism, whose content planning performance is comparable with ar models. by changeing the iteration number, pts can balance the generation quality and inference efficiency for various practical application requirements.