Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control (2023)

Abstract: Pretrained language models have demonstrated extraordinary capabilities in language generation. However, real-world tasks often require controlling the distribution of generated text in order to mitigate bias, promote fairness, and achieve personalization. Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions. However, many important distributions, such as personal preferences, are unquantified. In this work, we tackle the problem of generating text following arbitrary distributions (quantified and unquantified) by proposing NANO, a few-shot human-in-the-loop training algorithm that continuously learns from human feedback. NANO achieves state-of-the-art results on single topic/attribute as well as quantified distribution control compared to previous works. We also show that NANO is able to learn unquantified distributions, achieves personalization, and captures differences between different individuals’ personal preferences with high sample efficiency.

## introduction recent developments in large language models @xcite have advanced the state of automated text generation. however, to apply them to real-world tasks, it has become increasingly desirable to reduce social bias exhibited in large language models @xcite , improve fairness @xcite , and fit to diverse individual preferences @xcite . these desired properties are only defined over a set of generated text instead of individual sentences. therefore, they require control over the distribution of generated text @xcite . ## nano in general, controllable text generation operate on either an existing corpus of training examples or a description of the desired attribute or distribution. in this work, however, we adopt an active learning paradigm wherein human(s) can guide the model towards the desired attribute or distribution, allowing controlled generation with minimum manually written examples. the outer loop of nano is a "generate-feedbacktrain" loop. in each iteration of the loop, a number of samples are generated from what the model has learned so far (i.e. the model approximates p (x m+1:n |a, x 1:m ) as closely as possible). the generated samples are given to a human annotator, who rates the samples according to how accurately each conforms to the desired attribute or distribution. in addition, the human annotator can manually add new samples when the dataset lacks satisfactory samples. we keep the number of manuallyadded samples to a minimum (with a maximum of 5 added samples) while significantly reducing the number of rated samples in order to demonstrate our method's ability to self-improve with little human effort. finally, the model is trained on the labeled dataset and the trained model is used for generating text in the next iteration. in the following subsections, we detail each component of the outer loop. ## experiments and results in the following experiments, we demonstrate the ability of nano to generate text to (1) follow unquantified distributions and personalize, (2) follow quantified distributions, and (3) follow a single topic or attribute. ## conclusion in this work, we introduce nano, an algorithm that allows distribution control for text generation via few-shot human-in-the-loop training. we show that nano achieves better distribution control compared to previous works on both singletopic and quantified distributions with simple feedback from the human trainer, and demonstrate the ability of nano to efficiently fit its generation towards unquantified distributions and personal preferences. limitations: despite these successes, our current work is still limited in the following ways, which we leave to future work: