Active Learning for Natural Language Generation (2023)

Abstract: The field of Natural Language Generation (NLG) suffers from a severe shortage of labeled data due to the extremely expensive and time-consuming process involved in manual annotation. A natural approach for coping with this problem is active learning (AL), a well-known machine learning technique for improving annotation efficiency by selectively choosing the most informative examples to label. However, while AL has been well-researched in the context of text classification, its application to NLG remains largely unexplored. In this paper, we present a first systematic study of active learning for NLG, considering a diverse set of tasks and multiple leading selection strategies, and harnessing a strong instruction-tuned model. Our results indicate that the performance of existing AL strategies is inconsistent, surpassing the baseline of random example selection in some cases but not in others. We highlight some notable differences between the classification and generation scenarios, and analyze the selection behaviors of existing AL strategies. Our findings motivate exploring novel approaches for applying AL to generation tasks.

## introduction active learning (al) @xcite is a wellknown machine learning approach for reducing annotation effort, aiming to train models with less data by selecting the most informative examples to label. this paradigm was introduced and developed in the context of classification @xcite , and has been successfully applied to machine learning problems from a wide range of domains, including computer vision @xcite @xcite ) and text classification @xcite @xcite . major advances in the architecture and scale of machine learning models in general, and pretrained language models in particular, have given rise to the emerging field of natural language generation (nlg) @xcite . however, a major practical barrier in tackling nlg tasks is the shortage of annotated data, exacerbated by the increased burden of human annotation for such tasks. as a paradigm for minimizing labeling effort, al is a natural avenue for coping with these challenges. nevertheless, it has hardly been studied in the context of nlg problems. in this work, we explore the al paradigm as applied to nlg. our aim is twofold: first, to examine how well al strategies perform in nlg; and second, to gain insight into how these strategies operate in the unique context of text generation. to this end, we conduct an extensive set of experiments with leading al strategies on various nlg tasks, accompanied by a comprehensive analysis of strategy behaviour. our results reveal that these al strategies do not perform consistently across different datasets and tasks (figure 1 ), suggesting that new al methods are required to bring value in this domain. to the best of our knowledge, this is the first systematic study of al for nlg tasks. moreover, in this work we introduce strong instruction-tuned models into the al paradigm. in outlining the behavior of existing strategies, we aim to lay the groundwork for future research, leading to effective use of al in practical nlg scenarios. ## related work in the field of natural language processing, al was mainly studied for text classification @xcite @xcite . al has also been successfully applied in the context of neural machine translation (nmt), where the focus is on low-resource pairs of languages @xcite . some works investigated strategies that are tailored specifically for nmt, such as using a backward translator to check round-trip translations @xcite or quality estimation (chimoto and bassett, 2022). @xcite conducted a systematic comparison of different al methods in the context of nmt. thus, we do not include nmt in the present work and focus on nlg tasks that had not been systematically explored. there exists a wide variety of nlg tasks, and these have attracted much attention in recent years @xcite . nevertheless, outside the context of nmt there are very few works that apply al to nlg tasks @xcite . specifically, for summarization, @xcite propose a bayesian strategy in which they select samples to label by optimizing the uncertainty using the monte carlo bleu variance metric. more recently, @xcite propose an embedding-based method and show improvements in certain summarization datasets. paraphrase generation with lstms was reported in karaoguz (2018), where the authors use n-gram coverage measures as their sampling strategy, aiming at capturing the informativeness of source paraphrases. thus, while there have been some focused studies of al in the context of a specific nlg task, in this work we aim for a more systematic and comprehensive view, across multiple tasks and datasets. ## experimental setup this work aims to systematically study the application of active learning to nlg. to this end, we conduct a comprehensive set of experiments, exploring the behavior of the different families of al strategies across a wide range of nlg tasks. ## results for each dataset, we report performance metrics across 18 al iterations -i.e., the performance of models fine-tuned on different amounts of labeled instances -comparing results between different active learning strategies. in addition, we report the performance when using the baseline of random selection -randomly sampling the batch of examples to be added to the labeled data pool at each iteration. figure 2 depicts al results for two of the datasets tested, parabank1.0 and gyafc-f&r. as can be seen in the figure, for these datasets we do not see a clear advantage to any of the al strategies tested; moreover, the various al strategies do not seem to outperform the baseline of randomly selecting instances for labeling. these plots are representative of the overall pattern of results we see across tasks and datasets, with none of the al strategies convincingly overtaking the others across datasets. individual plots for all the datasets tested are shown in appendix a.3. a recurring pattern in the results is that most of the performance gains from supervision occur in the first few al iterations, and at times even in the first al iteration, where the model is trained on just 20 labeled instances. thus, it appears the flan base model needs only a small number of examples to learn the basic premise of the target generation task; while larger amounts of labeled data are of course beneficial, the few-shot performance of this model across datasets is quite strong. in figure 3 we present the full distribution of the performance of the various al strategies relative to the random selection baseline, for each of the 4 nlg tasks. overall, across tasks, datasets, al iterations and experiment repetitions, the behavior of all the strategies tested is strikingly similar to that of random selection. granted, there are specific instances where a strategy outperforms random selection for a specific dataset (for example, see the mc dropout strategy over the reddit tl;dr data); however, we find no consistent pattern of benefits to using al, even within a specific nlg task. to better quantify these results, we perform a wilcoxon signed-rank test, comparing each strategy to the random selection baseline (refer to appendix a.4 for details). figure 1 shows the results of the significance testing. as can be seen, none of the strategies exhibit a clear benefit for the tasks of paraphrase generation and formality transfer, and results for question generation and summarization are somewhat mixed. notably, both core-set and mc dropout fail to provide benefits across more than one dataset. ## analysis given the failure to systematically outperform the random baseline, in this section we examine the different al strategies by the properties of the batches they select, aiming to gain insights into how they operate. in line with the two families of strategies we survey, our analyses focus on notions of representativeness ( §6.1) and uncertainty ( §6.2). we perform a comparative analysis of the strategies, using the random strategy as a reference point. this analysis also serves as a sanity check, to validate that the properties exhibited by the different strategies are aligned with their expected behavior. to ensure all strategies are compared with the same initial conditions, this analysis is performed solely for the first al iteration, where all strategies rely on the same base model and the same unlabeled set u d . each strategy makes its own selection of 100 examples foot_2 for labeling from u d . ## discussion in this work, we examined the effectiveness of various active learning strategies across multiple nlg tasks. through rigorous experimentation and analysis, we have shown that no al strategy systematically demonstrates a clear superiority over the random baseline in terms of nlg quality. our findings indicate that despite the potential promises and advancements in active learning techniques, when it comes to nlg, the inherent complexity and diversity of human language poses significant challenges. al strategies, which aim to improve efficiency by actively selecting informative data points for labeling, may not effectively capture the intricacies of language structure, semantics, and context required for generating coherent and meaningful text. our results provide a wider and somewhat contrasting perspective to previous works. while previous papers had typically reported the effectiveness of al on a specific dataset and task, our comprehensive analysis -spanning multiple datasets and tasks -suggests that the potential gains reported before are not very robust, hence do not show up in a consistent manner. thus, while our experiments were done on a single base model and hyperparameter setting, they indicate that existing al methods cannot be assumed to work out-of-the-box. the failures of existing al methods for nlg cannot easily be associated with a single underlying factor. more likely, there are multiple issues at play that violate some of the premises and assumptions of al strategies that were designed with classification in mind. some of these potential causesfor instance, the complex relation between model uncertainty and erroneous outputs -reflect a fundamental difference between the tasks of classification and generation. others, however, may be more a question of degree. for instance, the output space for generation is overwhelmingly larger than that of a typical classification task, and is also characterized by a large degree of label imbalance, properties that may lead to difficulties in capturing informative examples. notably, similar issues exist in some multi-label classification tasks, which also exhibit difficulties with leveraging al @xcite . in this work we combine al with a strong instruction-tuned model, highlighting the importance of the base model used. the behavior observed here, where much of the performance gain is achieved with a small number of training examples, is encouraging with respect to the practical scenario of a limited labeling budget; at the same time, this may entail new al approaches that focus on small batches or on bringing value to the selection of a single batch of few-shot instances. in our view, the takeaway from our results is not that the paradigm of active learning should be abandoned. the goal of reducing manual annotation efforts remains as relevant as ever, all the more so for the particularly expensive annotation process associated with nlg. rather, our hope is that these results will stimulate new efforts in devising novel al strategies and approaches, ones that are specifically designed for the nlg scenario, and suited for strong instruction-tuned base models.