Learning Natural Language Generation with Truncated Reinforcement Learning (2022)

Abstract: This paper introduces TRUncated ReinForcement Learning for Language (TrufLL), an original approach to train conditional languagemodels without a supervised learning phase, by only using reinforcement learning (RL). As RL methods unsuccessfully scale to large action spaces, we dynamically truncate the vocabulary space using a generic language model. TrufLL thus enables to train a language agent by solely interacting with its environment without any task-specific prior knowledge; it is only guided with a task-agnostic language model. Interestingly, this approach avoids the dependency to labelled datasets and inherently reduces pretrained policy flaws such as language or exposure biases. We evaluate TrufLL on two visual question generation tasks, for which we report positive results over performance and language metrics, which we then corroborate with a human evaluation. To our knowledge, it is the first approach that successfully learns a language generation policy without pre-training, using only reinforcement learning.

## introduction since the development of generic language models trained on massive unlabelled text corpora @xcite , state-of-the art language processing systems rely on sequential transfer learning @xcite . the pretrained language model (lm) is fine-tuned on the downstream task using a standard supervised learning (sl) 1 code is available at https://github.com/ amdonati/rl-nlp vqg, trufll truncates the vocabulary space by using a language model. here, 'run,' and 'the' are syntactically incorrect and thus truncated. yet, 'car' is not trimmed as the lm is not visually grounded. (right) in a vqg training loop, the agent generates a question given an image-answer pair, which is then fed to a vqa model predicting an expected answer. if both answers match, the agent is rewarded. on the other hand, text generation can be naturally framed as a sequential decision making problem, with the sequence of words seen as successive actions over a vocabulary. thus, some researchers have recently focused on learning language models using instead reinforcement learning (rl) @xcite @xcite . rl methods allow acquiring language through interactions within rich and diverse environments @xcite , help understanding language acquisition and language pragmatics @xcite . "reward is enough" @xcite highlights the necessity of using rl for ai systems to acquire language in its full richness. indeed, (i) language may be intertwined with other modalities of action and observation, (ii) the utility of language varies according to situations and behaviours, (iii) it is consequential and purposeful, and (iv) some linguistic problems are better solved dynamically, through experience (such as using a diplomatic tone in a speech.) in addition, rl allows optimizing a non-differentiable learning signal, hence handles more diverse objective functions, and also avoids some of the text degeneration issues previously mentioned. so far, rl-based text-generation tasks have relied on a pre-training phase to ease learning: the policy language model is trained with sl on the task dataset, before being fine-tuned with policy gradient methods @xcite on the task at hand. those approaches often require human-labelled datasets. besides, combining pre-training and fine-tuning phases either barely change the policy distribution, or induces language drift @xcite , i.e the generated language drifts semantically or syntactically from natural language. in this paper, we aim at learning a conditional language model using rl without a pre-training phase, so that (i) we get free from datasets with human annotations, and (ii) we avoid the text generation flaws induced by the common methods. while appealing, such an approach requires overcoming the hurdle of the combinatorial language action space, a vocabulary usually containing more than 10,000 words. yet, while large and discrete, a language action space contains a specific structure, made of all the syntactical and semantics rules of a given language. trufll leverages such structure to drive the exploration of the rl-based language agent during training. at each time step of the text generation process, trufll truncates its effective action space to a small subset of words provided by a pretrained task-agnostic language model. such an approach injects a generic prior linguistic knowledge into the rl algorithm, is usable on tasks lacking in-domain labeled data, and can be easily transferred to new rl-based text generation tasks. thus, trufll can be applied to any language generation task given a generic lm and a reward. we here evaluate it on two visual question generation (vqg) tasks, the synthetic clevr dataset @xcite , and the natural language vqav2 dataset @xcite . unlike alternative rl without pre-training approaches, trufll manages to ask meaningful and valid questions on large vocabularies, exhibiting success rate and language metrics close to pretrain models with labeled data, while producing more original language. formally, a language generation agent is defined by a policy π θ (a distribution over v) parametrized by θ, first initialized with the context c. at each time step t, the agent samples a new word w t from its policy π θ (w t |@xmath0, c). it moves to a new state (@xmath1, c) and receives a reward r @xmath2,c,w t ), where r is a reward function relative to the language task. the rl language agent aims to learn a policy that maximizes e π θ [ t @xmath3, foot_0 while generating the sequence of words @xmath4, where e π θ is the expectation under π θ , and t the maximal length of the words sequence. policy gradient this optimization process may be performed through policy gradient (pg) algorithms @xcite . in the language literature, reinforce @xcite has been used as a simple monte carlo approximation of this gradient @xcite .yet, in this paper, we use a proximal policy optimization approach (ppo) @xcite to have a lower variance and better convergence rate; ppo clips the gradient estimate to have smooth policy updates. for all 0≤@xmath5, let s @xmath6,c) and a @xmath7. policy gradient methods minimize the objective: ppo then keeps track of the previous policy π θ old before the pg update to compute the training objective: secondly, the loss is completed with an entropy term to soften the policy distribution, which computes for each timestep t of an episode h(π θ (a t |s t )), where h is the entropy function. ## trufll we here aim at making rl methods feasible in the language setting by dynamically reducing the action space, i.e., by restricting the language agent to select a word within a subset of the vocabulary at each time step. we detail below the action space's truncation model and the associated rl algorithm to learn the language agent. ## experimental setting we here list the experimental setting and detail the network and hyperparameters in appendix a.4. ## conclusion we proposed trufll, an original approach to learn a natural language generation (nlg) task using rl, without the usual pre-training phase requiring supervised datasets. to our knowledge, this is the first rl-based algorithm dedicated to learning a word-based text-generation task, which does not rely on a pre-training phase while scaling to large vocabularies. although it comes with its limitations, the truncated rl algorithm provided by trufll gets free from labelled data in task-oriented language models, presents interesting language generation properties, and provides a generic and transferable method to learn any nlg problem.