Hierarchical Recurrent Aggregative Generation for Few-Shot NLG (2022)

Abstract: Large pretrained models enable transfer learning to low-resource domains for language generation tasks. However, previous end-to-end approaches do not account for the fact that some generation sub-tasks, specifically aggregation and lexicalisation, can benefit from transfer learning in different extents. To exploit these varying potentials for transfer learning, we propose a new hierarchical approach for few-shot and zero-shot generation. Our approach consists of a three-moduled jointly trained architecture: the first module independently lexicalises the distinct units of information in the input as sentence sub-units (e.g. phrases), the second module recurrently aggregates these sub-units to generate a unified intermediate output, while the third module subsequently post-edits it to generate a coherent and fluent final text. We perform extensive empirical analysis and ablation studies on few-shot and zero-shot settings across 4 datasets. Automatic and human evaluation shows that the proposed hierarchical approach is consistently capable of achieving state-of-the-art results when compared to previous work.

## introduction the recent development of large pretrained language models (plms; i.e. bert @xcite , gpt-3 @xcite , t5 @xcite ) has caused a shift of interest in the research community towards domain adaptation and transfer learning. for the task of concept-totext natural language generation (nlg), wherein the aim is to generate a natural language text that describes the semantic content of an abstract structured machine-readable input (meaning representation; mr), transfer learning from plms has become a popular and high performing approach with 13 out of the 15 participating teams in the latest webnlg+ shared task @xcite employing a fine-tuned pretrained model as their main submitted system. specifically, t5-based systems achieved a human evaluation ranking on par with the ground truth in terms of fluency and adequacy. transfer learning from plms also enables training on few-shot and zero-shot settings, i.e. when sufficient in-domain data are unavailable. prominent and relevant examples include machine translation @xcite and nlg for task-oriented dialogues @xcite . this paper focuses on concept-to-text nlg, where recent machine learning and in extension transfer learning approaches adopt an end-to-end architecture @xcite ) that inputs the full meaning representation and produces the full output text. in such end-to-end models, the traditional sub-tasks @xcite involved in language generation (i.e. planning, lexicalisation, aggregation, referring expression generation, and surface realisation) are performed implicitly. however, we posit that some of these sub-tasks, specifically lexicalisation (i.e. choice of vocabulary) and aggregation (i.e. process of combining simpler sentence structures to form complex ones), exhibit varying potential for exploiting transfer learning as the former is more domain-specific than the latter. for example, it is more difficult to exploit transfer learning for lexicalisation since if certain words are not already associated with a particular mr input, fewshot learning may not be able to create a strong association through the limited data. this is further exacerbated in zero-shot learning. on the other hand, the knowledge required to form complicated sentence structures and apply aggregation strategies is more commonly shared between domains and would benefit more from transfer learning. we aim to exploit these differing potentials for transfer learning in few-shot and zero-shot generation, via a new hierarchical approach to conceptto-text nlg. specifically, we propose hierarchical recurrent aggregative generation (hrag), a three-moduled architecture where the first module is in charge of independently lexicalising each unit of information in the input as a sub-phrase (e.g. a phrase expressing that unit of information alone), the second module is responsible for recurrently aggregating these sub-units to generate a unified text, and the third module rephrases it to produce a coherent and fluent output; see figure 1 . these are jointly trained via a loss that combines their discrete objectives. concept-to-text is ideal for hrag as mrs can be split into attribute-value pairs that vaguely correspond to output sub-phrases. in this paper, we (i) present hierarchical recurrent aggregative generation and experimentally demonstrate the benefits of separately applying transfer learning to language generation subtasks; (ii) facilitate the model's training by inferring module-specific training signal from the available output targets; (iii) provide extensive empirical analysis and ablation studies on few-shot and zero-shot settings across 4 datasets, one of which we adapt ourselves for few-shot learning; (iv) perform human evaluation comparing our proposed approach to previous work on few-shot generation. our automatic and human evaluation results show that our hierarchical approach achieves state-of-theart results when compared against previous work. ## method figure 1 shows the overall structure of the proposed hierarchical model hrag. its three modules are in charge of lexicalisation, aggregation and postedit, and are inspired by traditional nlg stages and their specific potential for transfer learning in a few-shot setting. figure 2 shows an example of how the outputs of each stage are formed. ## related work despite being an important research topic with real-life applications, domain adaptation for lowresource/few-shot concept-to-text nlg has not been extensively researched. wen et al. (2016) leveraged the scarcity of target in-domain data by augmenting it with synthetic data, tran and nguyen (2018) used variational autoencoders in conjunction with text similarity and domain critics to better guide the fine-tuning process, while mi et al. (2019) tackled the problem by defining domain adaptation as an optimisation meta-learning task. most recently, peng et al. (2020) and kale and rastogi (2020) have proposed the use of pretrained language models to tackle few-shot and zero-shot learning in concept-to-text nlg, achieving significant gains over strong non-pretrained baselines. specifically, peng et al. (2020) proposed sc-gpt, a semantically conditioned gpt-2 model, wherein, prior to few-shot learning, the gpt-2 model is further fine-tuned on a number of task-oriented dialogue datasets in order to mitigate the problem of representation bias. on the other hand, in kale @xcite , a set of human-authored templates are used to generate high-quality sentences corresponding to each unit of information in an mr. these are then concatenated and given as input to a t5 model (t2g2) to form a coherent sentence. in this paper's evaluation, we opt to compare our approach against the naive t5 baseline introduced by @xcite , as it is shown to overly outperform sc-gpt by basically replacing the underlying gpt-2 model for t5, and sc-gpt was outperform all previous non-pretrained baselines. we do not compare against t2g2, as access to human authored templates or other such manually annotated resources, which are by nature very domain-specific and costly to create, are not necessarily guaranteed in low-resource settings. we note that t2g2 is equivalent to the naive t5 when templates are not employed. in our proposed system, the hierarchy emerges from modelling the lexicalisation and aggregation sub-tasks on separate layers. previous attempts in exploring hierarchical structures for text generation tasks instead focused on modelling different aspects of the input or output. in concept-to-text nlg for task-oriented dialogues, @xcite proposed a multi-layered decoding process where each layer was responsible for generating words associated with specific part-of-speech tags. @xcite and @xcite took advantage of the intrinsically hierarchical structure of dialogue acts to create better input representations and ease domain adaptation. our approach is also related to coarse-to-fine approaches, which have been explored in story @xcite , review @xcite and keyphrase @xcite generation tasks. however, in these tasks, the output is not necessarily restricted to be an exact realisation of the input, and can be initially loosely prompted or drafted, and subsequently expanded. ## conclusion we proposed hierarchical recurrent aggregative generation, a three-moduled jointly trained archi-tecture, designed to exploit the different extents to which lexicalisation and aggregation can benefit from transfer learning. additionally, due to the lack of explicit training signals for hrag's modules, we show how module-specific targets can be inferred from the available output targets. extensive automatic metric experiments and analysis across 4 datasets, as well as accompanying human evaluation, demonstrates that hrag outperforms previous state-of-the-art approaches, especially in regards to missing slot error and adequacy. language technologies, pages 120-129, san diego, california. association for computational linguistics. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, remi louf, morgan funtowicz, joe davison, sam shleifer, patrick von platen, clara ma, yacine jernite, julien plu, canwen xu, teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. xiaoxue zang, abhinav rastogi, srinivas sunkara, raghav gupta, jianguo zhang, and jindong chen. 2020. multiwoz 2.2 : a dialogue dataset with additional annotation corrections and state tracking baselines. in proceedings of the 2nd workshop on natural language processing for conversational ai, pages 109-117, online. association for computational linguistics. giulio zhou and gerasimos lampouras. 2021. generalising multilingual concept-to-text nlg with language agnostic delexicalisation. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 114-127, online. association for computational linguistics. barret zoph, deniz yuret, jonathan may, and kevin knight. 2016. transfer learning for low-resource neural machine translation. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 1568-1575, austin, texas. association for computational linguistics.