Most NLG is Low-Resource: here’s what we can do about it (2022)

Abstract: Many domains and tasks in natural language generation (NLG) are inherently ‘low-resource’, where training data, tools and linguistic analyses are scarce. This poses a particular challenge to researchers and system developers in the era of machine-learning-driven NLG. In this position paper, we initially present the challenges researchers & developers often encounter when dealing with low-resource settings in NLG. We then argue that it is unsustainable to collect large aligned datasets or build large language models from scratch for every possible domain due to cost, labour, and time constraints, so researching and developing methods and resources for low-resource settings is vital. We then discuss current approaches to low-resource NLG, followed by proposed solutions and promising avenues for future work in NLG for low-resource settings.

## introduction natural language generation (nlg) is the process of generating text from structured or unstructured data and has recently received renewed attention due to the emergence of large pre-trained language models (e.g. @xcite ) that promise to generate output that is more natural, variable, and adaptable to new domains as compared to rulebased approaches. however, the development of robust, controllable, and usable nlg systems depends heavily on the availability of large, highquality, labelled datasets that are appropriate for the task at hand @xcite . unfortunately, for most domains such data is unavailable, probably with the exception of weather, restaurant, and sports domains. even in the aforementioned domains, data is fairly small compared to lowresource tasks & languages in other areas, such as in machine translation. in this paper, we focus on controllable nlg tasks that can be framed as data-to-text generation tasks, rather than language prediction models. we ## analyses & tools while the primary data for nlg are parallel corpora, a variety of other resources can facilitate development. language models (lms) can approximate fluency measures and can be used to produce texts using sampling @xcite . linguistic analyses provide insight into what makes a text well-formed and assist in the design of rulebased systems and architectures for ml-based approaches. finally, tools & resources such as parsers, part-of-speech taggers, semantic role labellers, ontologies, morphological analysers, and word embeddings can help to decompose nlg subtasks and make the problem more approachable. ## mitigation strategies so far we have discussed the corpora, analyses, & tools that are often missing in low-resource settings. we now turn to mitigation strategies for dealing with this lack of resources, namely data augmentation, transfer learning, meta-learning, feedbackbased learning, and rule-based methods. ## discussion & promising directions so far, we have highlighted the ways in which nlg is influenced by the availability of data, analyses, and tools at both the language and the domain level and described a number of trainable approaches which aim to learn an nlg system given different resource limitations. it is clear that it is neither sustainable to collect large aligned datasets for every domain nor use general purpose language models off-the-shelf for controlled nlg, so researching and developing approaches for low-resource settings that distill the strengths of pre-trained language models while focusing on controllable approaches is vital. here we outline some promising directions in this area. ## conclusion in this paper we have highlighted ways in which nlg is often low-resource in data, analyses, or tools with respect to either the target language or domain. the number of corpora for data-to-text generation remain limited, but there are a number of strategies for adding meaning annotations to existing texts, or vice versa, and crowdsourcing new data sets. large language models are also not always available: they are typically only available for relatively high-resource languages. linguistic analyses and automated analysis tools are more widely available and can help decompose the generation task to make it easier to develop systems in low-resource settings. fortunately there are a variety of mitigation strategies available, including methods for data augmentation, transfer learning, meta-and feedback-based-learning, and different ways of structuring the task. this makes for an interesting set of challenges and an exciting set of opportunities to improve the state-of-the-art for natural language generation in low resource settings. our own current work is beginning to explore some of these promising directions, including multi-task learning, pipeline approaches to nlg, and transfer learning from related languages and general purpose llms. however, there is much work to be done, and we hope the community embraces these challenges.