Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach (2022)

Abstract: Although remarkable progress on the neural table-to-text methods has been made, the generalization issues hinder the applicability of these models due to the limited source tables. Large-scale pretrained language models sound like a promising solution to tackle such issues. However, how to effectively bridge the gap between the structured table and the text input by fully leveraging table information to fuel the pretrained model is still not well explored. Besides, another challenge of integrating the deliberation mechanism into the text-to-text pretrained model for solving the table-to-text task remains seldom studied. In this paper, to implement the table-to-text generation with pretrained language model, we propose a table structure understanding and text deliberating approach, namely TASD. To be specific, we devise a three-layered multi-head attention network to realize the table-structureaware text generation model with the help of the pretrained language model. Furthermore, a multi-pass decoder framework is adopted to enhance the capability of polishing generated text for table descriptions. The empirical studies, as well as human evaluation, on two public datasets, validate that our approach can generate faithful and fluent descriptive texts for different types of tables.

## introduction the task of learning to generate natural language descriptions from non-linguistic input, which is referred to as data-to-text, is important for many applications, such as weather forecast generation @xcite , sports news writing @xcite , biography writing @xcite , market comments writing @xcite and automatic question-answering @xcite . the input data can be in various forms for data-to-text though, here we focus on the text generation task that takes the table as input. inspired by neural machine translation models, previous studies on table-to-text tasks mainly adopt traditional seq2seq methods to generate table descriptions @xcite @xcite @xcite . despite generating text with high fluency, lacking numerous source tables leads to lower generalizability of the tableto-text model. recent progress in the pretrained language model @xcite shows remarkable performance in solving natural language processing tasks. the model pretrained on large-scale data possesses rich knowledge, which inspires us with the potential for solving generalization issues of the text generation task. to exploit the expressive power of the pretrained model for the table-to-text task, it is necessary to serialize the input table effectively. several works have put efforts to bridge this gap, such as serializing the table into a token sequence @xcite @xcite , or introducing an extra task to control the table representation @xcite . however, none of these leveraged the table structure information effectively. furthermore, the text-to-text pretrained model decodes and generates a sequence in a onepass forward process, which means it cannot perceive the future words in advance on the target side. recently, the deliberation mechanism @xcite implemented by the multi-pass decoder is proposed to tackle this problem. however, how to adapt this approach for text-to-text pretraining, which can be further applied to the table-to-text task, is another challenge. to this end, we propose a table structure understanding and text deliberating approach, namely tasd, to solve the table-to-text task with the pretrained language model enhanced by the deliberation mechanism. specifically, we first serialize the table input with customized templates which do not acquire the target cells to be labeled. then, we employ the multi-head attention in a hierarchical way to learn the table representation that is aware of table structure and apply it to guide the fine-tuning of the text-to-text pretrained model. afterward, we adopt the multi-pass decoder to realize text deliberation. more specifically, we treat the above table-structure-aware fine-tuned model as the first-pass decoder and adopt another pretrained model as the second-pass decoder to further polish the descriptive text. in the second-pass decoding phase, the table representation can be conveniently leveraged as the "original text" in the text deliberation mechanism. the main contributions of this work can be summarized as follows: and we develop an effective text deliberation method dedicated to the table-to-text task. fine-tuning the model pretrained in a large corpus and adapting to a specific task is an effective approach to tackling the generation issues disturbed by small data and large parameters @xcite . @xcite ) explored the feasibility of applying the text-to-text pretrained model to the table-to-text task, @xcite applied multi-task learning to solve the table-to-text task with pretrained language model, and ( suadaa et al., 2021 ) leveraged pretrained language model for fact inference in numerical table contents. however, these approaches seldom perceived and integrated the complete table information into the fine-tuning of the pretrained model. a table-to-text pretrained model @xcite was proposed though, the large and diversified table corpus is often unavailable. in addition, recent works on fact verification taking tabular as input @xcite have suggested the effectiveness of the table-structure-aware pretrained model. ## methodology in this section, we introduce the proposed framework in detail. as shown in fig. 1 , our framework mainly consists of three components, i.e., templatebased table serialization, table-structure-aware fine-tuning, and text deliberation. specifically, we first produce a sequence describing the table contents with customized templates. the templates we adopted do not require the target cells to be labeled. then, to generate informative text, we adopt full table representation learning to guide the description generation, such that the outcome text is capable of emphasizing and delineating the facts in the table from a macroscopic perspective. finally, we employ and adapt the multi-pass decoder to our data-to-text problem, which can further fine-tune the generated table description. technical details for all three modules will be introduced separately in the following subsections. ## discussion in our work, we devised a two-pass decoder framework dedicated to the table-to-text task with the help of the table-structure-aware text generation model (i.e., tasatg). however, the effectiveness of the text deliberation for the table-to-text task should be further explored and integrated into the table-structure-aware modeling in a more harmonic table reconstruction for table-structure-aware modeling enhancement. manner. to discuss the limitation of the text deliberation of tasd, we additionally developed a table content reconstruction loss and integrate it into tasd in a multi-task learning fashion. specifically, given the table-structure-aware embedding e (2) generated with eq. ( 3 ), we randomly mask certain cells of the input table and yield a partially corrupted embedding of the input table, denoted by e (2) . then, a two-layer mlp (i.e., multilayer perceptron) is adopted to restore the tablestructure-aware embedding. afterward, an mse (i.e., mean square error) loss is adopted to measure the effectiveness of table reconstruction and further integrated into the tasd framework in the multi-task learning paradigm. the process of table reconstruction is demonstrated in fig. 5 . we carried out a series of experiments to evaluate the performance of tasd w/ and w/o the help of table reconstruction loss (i.e., trloss) on nu-mericnlg and totto datasets in terms of bleu-n (1 to 4), meteor, and rouge-l. the results can be found in tables 4 and 5 . according to the results reported on the nu-mericnlg dataset, the trloss is helpful in enhancing the capability of table comprehension though, the best performance is achieved by tasd w/o d w/ trloss . it seems that the performance improvement gained by the table comprehension enhancement is sacrificed after the text deliberation is adopted. meanwhile, on the totto dataset, tasd with the table reconstruction (i.e., tasd w/ trloss ) does achieve the best performance in terms of bleu-1, bleu-2, meteor, and rouge-l, though the improvement is not remarkable. the contents of the input tables are mainly linguistic and the table structures are not too diverse might be able to explain the performance improvement of tasd w/ trloss on the totto dataset. with the above comparisons, we can conclude that, for the input tables with diverse structures, the limitation of the current text deliberation mechanism cannot be neglected if one aims to enhance the capability of table comprehension for the table-to-text task. moreover, this also suggests that the generalization capability of text deliberation of tasd should be improved in the future. limitations. in this work, long tables in the totto dataset are removed since the efficiency and performance of tasd on large tables could be lowered. in the future, the capability of handling long tables for table-to-text models should be further explored. besides, a large-scale and more exhaustive human evaluation is necessary. we plan to recruit more volunteers to conduct the human annotation. ## conclusion in this paper, to realize table-to-text with the pretrained language model, we proposed a table structure understanding and text deliberating approach, namely tasd. the table structure understanding was realized by developing a hierarchical multihead attention network, which can benefit the finetuning of the text-to-text pretrained model. the fully represented table information benefits not only the pretrained language model but also the text deliberation process since the structure information with rich semantics could be fed into the second-pass decoding naturally. we carried out extensive experiments on two public datasets with different table types. automatic and human-based evaluations, as well as qualitative analysis, validate the effectiveness of our approach to generating faithful and fluent table descriptions. in the future, we will improve text deliberation by devising a unified framework to integrate the multi-pass decoder and refine the descriptive text paying more attention to sentence fluency.