Improving User Controlled Table-To-Text Generation Robustness (2023)

Abstract: In this work we study user controlled table-to-text generation where users explore the content in a table by selecting cells and reading a natural language description thereof automatically produce by a natural language generator. Such generation models usually learn from carefully selected cell combinations (clean cell selections); however, in practice users may select unexpected, redundant, or incoherent cell combinations (noisy cell selections). In experiments, we find that models perform well on test sets coming from the same distribution as the train data but their performance drops when evaluated on realistic noisy user inputs. We propose a fine-tuning regime with additional user-simulated noisy cell selections. Models fine-tuned with the proposed regime gain 4.85 BLEU points on user noisy test cases and 1.4 on clean test cases; and achieve comparable state-of-the-art performance on the ToTTo dataset.

## introduction the goal of table-to-text generation is to provide the user with a description of the most relevant content in a given table @xcite @xcite . recently, @xcite proposed a controlled table-to-text generation task where the goal is to automatically create a description for a determined subset of the table, namely the highlighted table cells. the main focus on parikh et al.'s 2020 work is to assess the performance of neural text generators in a more controlled setting, i.e., when given an input table with explicit instructions (i.e., highlights) on what should be expressed in the output description. in this work, we view this task in the context of a natural language interface, as a user controlled tableto-text generation task, where users provide those figure 1 : an example in the totto dataset. the figure is retrieved from @xcite . the cells coloured in yellow are the highlight cells. a crucial aspect of usability assessment for a generator in this interactive table-to-text task is robustness. in a recent study by @xcite , it has been shown that neural generation models fail to maintain their in distribution performance when confronted with realistic scenarios at test time such as typos in the input text. in the case of user controlled table-to-text generation, users may introduce noise when exploring the table content and select cell combinations that turn out to be unexpected, redundant, or incoherent. for example, in figure 1 , when the user wants to express "eleven seasons", they might miss one year or highlight the header cell. they may also select unrelated headers, for instance adding the header "lng" to the current selection. existing controlled table-to-text generation models @xcite @xcite ) are trained on carefully selected cell combinations (clean cell highlights) from the totto dataset @xcite . we argue that these models will not generalize well in practice with user noisy highlights. no previous work has study model robustness under this practical set up. we carry out a usability study to observe how users highlight cells in a table. based on the imperfect cell selections that users produce, we automatically create additional data examples by corrupting examples from the original totto dataset. we then fine-tune state-of-the-art table-to-text neural generation models with this additional data. we compare the performance of models fine-tuned only with clean cell highlights versus those trained with additional noisy cell highlights, both on a test set with clean and noisy highlights. experimental results show that models fine-tuned with clean cell highlights only perform well on clean test cases (i.e., performance drops dramatically when evaluated on noisy cell highlights). that is, these models do not generalise well in practice with user noisy cell selections. in contrast, the proposed training scheme with additional noisy cell highlights not only makes user controlled table-to-text models achieve better performance in practical scenarios, but it also boosts performance on perfect inputs. experimental results show that models fine-tuned with our proposed training regime gain 4.85 bleu points on noisy and 1.4 bleu points on clean highlights; and achieve comparable state-of-the-art performance on the totto dataset. 2 ## methodology we describe the process for creating user noisy cell highlights from examples in totto @xcite ) ( §2.1 and §2.2). then, we evaluate models optimized with the standard training scheme (i.e., only on clean cell highlights) on the created noisy test cases. results show that these models perform poorly. to improve model robustness, we propose a new learning regime described in §2.3. to further improve performance, we fine-tune with reinforcement learning (rl) based optimisation ( §2.4). finally, §2.5 summarises the learning schemes and objective functions we propose for robust user controlled table-to-text generation. ## experimental results implementation details for our table-to-text generation models can be found in appendix b. we use the same hyperparameters as the baseline in the totto @xcite . as shown in in contrast, we observe that our proposed learning scheme makes generators achieve better performance both on clean and noisy cell selections. on reference: in 2005, kosuke matsuura again drove for super aguri fernandez racing, and again finished 14th with a best place finish of 6th in the two races. ours: in 2005, kosuke matsuura drove for super aguri fernandez racing in the indycar series and finished 14th in points. baseline: in 2005, kosuke matsuura drove for super aguri fernandez rac--ing and finished 14th in the wgl 6 and 280 points. reference: the last asian beach games was held in danang, vietnam from 24 september to 3 october 2016, while the next will be held in 2020 in sanya, china, the first to breakaway from the 2-year cycle. ours: the asian beach games are scheduled to be held in da nang, vietnam from september 24 to october 3, 2016 and in sanya, china in 2020. baseline: the asian beach games were held from 2014 to 2016 in da nang, vietnam and from 3 october to 3 october 2020 in sanya, china. list of rulers of brittany iain glein section title : house of montfort section title: awards and nominations name birth death year title award category result peter ii the simple (pêr ii) 1450-1457 7 july 1418 22 september 1457 nantes aged 41 1990 silent scream silver bear best actor won arthur iii the justicier (arzhur iii) 1457-1458 24 august 1393 26 december 1458 nantes aged 65 reference: at the very end of his life, arthur iii became duke of brittany, succeeding peter ii. ours: arthur iii the justicier was duke of brittany from 1457 until his death in 1458, succeeding peter ii the simple. baseline: arthur iii (26 december 1458) was duke of brittany from 1450 to his death. model fl fa cc clean bart-large (clean) 0.83 0.83 0.89 bart-large + rl (d f inal ) 0.88 0.89 0.93 noisy bart-large (clean) 0.80 0.81 0.87 bart-large + rl (d f inal ) 0.89 0.91 0.91 in appendix a we provide additional experiments on ablation results on the contribution of each noise dataset, training with a subset of d f inal (i.e., training with one fifth of the data also improves robustness), and evaluating on cases with different amount of noise (i.e., our approach generalises better to cases with higher values of k). to gain insights on how the improvements are perceived in generated descriptions, we conduct a human evaluation. we follow the setup described in @xcite . we sample 100 development instances and have five human judges (voluntary msc level students fluent in english) to annotate them across three criteria. fluency (users select amongst fluent, mostly fluent, and not fluent; we report the percentage of outputs annotated as fluent; faithfulness (a candidate sentence is considered to be faithful if all the information in it is supported by the highlight cells and metadata of the table; we report the percentage of outputs that users annotate as faithful); and covered cells (the percentage of highlighted cells that the candidate sentence covers; we report average percentage of covered cells across all sampled instances). we choose the best performing model, bart-large + rl (d f inal ), fine-tuned with the proposed approach and compare it with state-of-theart models on the totto test set. these are ncp @xcite , pointer-generator @xcite , bert-to-bert @xcite , and t5-3b @xcite , lattice @xcite , and plangen @xcite . figure 2 shows two instances of synthetic noisy cell selections of type noise 1 (i.e., accidentally selected random cell not related to the exploration intention) and type noise 2 (i.e., random criteria for header selection); and two instances of user noisy cell selection from the human study of type noise 3 (i.e., highlight 2014 semantically close to cells in the exploratory intention) and noise 4 (i.e., won is not highlighted). cells in yellow indicate original highlights from the totto dataset and those in orange are noisy selections. in both cases, the outputs produced by the model fine-tuned with the proposed regime are not affected by noise and show better coverage, factual accuracy, and lexicalisation. this illustrates human evaluation preferences. ## conclusion we study the performance of user controlled tableto-text generation. we show that standard training schemes with only carefully selected cells causes poor robustness of generators in practice when confronted with user noisy cell selections. to address this, we introduce a training scheme with simulated user noisy cell selections. experimental results show that generators optimized with our proposed scheme can achieve better performance on both clean and noisy cell selections. in the future, it would be interesting to investigate how to apply our approach to other data-to-text datasets to improve model generalisation.