PPL-MCTS : C onstrained Textual Generation Through Discriminator-Guided MCTS Decoding (2022)

Abstract: Large language models (LM) based on Transformers allow to generate plausible long texts. In this paper, we explore how this generation can be further controlled at decoding time to satisfy certain constraints (e.g. being non-toxic, conveying certain emotions, using a specific writing style, etc.) without fine-tuning the LM.Precisely, we formalize constrained generation as a tree exploration process guided by a discriminator that indicates how well the associated sequence respects the constraint. This approach, in addition to being easier and cheaper to train than fine-tuning the LM, allows to apply the constraint more finely and dynamically. We propose several original methods to search this generation tree, notably the Monte Carlo Tree Search (MCTS) which provides theoretical guarantees on the search efficiency, but also simpler methods based on re-ranking a pool of diverse sequences using the discriminator scores. These methods are evaluated, with automatic and human-based metrics, on two types of constraints and languages: review polarity and emotion control in French and English. We show that discriminator-guided MCTS decoding achieves state-of-the-art results without having to tune the language model, in both tasks and languages. We also demonstrate that other proposed decoding methods based on re-ranking can be really effective when diversity among the generated propositions is encouraged.

## introduction generative language models exist for a long time, but with advent of the transformer architecture @xcite and increasing computing capabilities, they are now able to generate well written and long texts. in particular, large models, such as the well known gpt-2 @xcite and @xcite , have been used successfully for various applications: assisting writers, summarizing, augmentating data for subsequent nlp tasks, generating fake news @xcite @xcite . yet, beside the prompt used to initiate the generation process, there are few options to have control on the generation process. being able to add some constraints on the generated texts is useful for various situations. for example, it allows to create texts that follow a certain writing style, convey a certain emotion or polarity or to ensure that a generated summary contains correct information. more critically, it can be used to prevent the inherent toxicity of language models trained on the internet, or to not reproduce gender or race stereotypes. so far, most methods necessitate to fine-tune the lm, so that it specifically learns to model this constraint, i.e. the constraint is -hopefully-incorporated in the lm. this finetuning approach has several drawbacks. it implies to train multiple specific lms (one per constraint), which is costly, when even possible given the size of current state-of-the-art lm, and results in several models. in this paper, we propose new approaches to add such additional constraints on the texts but at decoding time. we exploit a discriminator that is trained to determine if a text follows a given constraint or not; its output provides information to guide the generation toward texts that satisfy this expected constraint. in order to make the most of the discriminator information, we propose an original method based on the monte carlo tree search (mcts) algorithm @xcite , namely plug and play language -monte carlo tree search @xcite . we also propose simpler methods based on re-ranking to fulfil this goal. both approaches do not require to fine-tune the lm; adding a new constraint can thus simply be done by providing a discriminator verifying if a text complies with what is expected. more precisely, our main contributions are the following ones: the goal of constrained textual generation is to find the sequence of tokens x 1:t which maximises p(x 1:t | c), given a constraint c. few methods address the constrained textual generation. class-conditional language models. classconditional language models (cc-lms), as the conditional transformer language (ctrl) model @xcite , train or fine-tune the weights θ of a single neural model directly for controllable generation, by appending a control code in the beginning of a training sequence. the control code indicates the constraint to verify and is related to a class containing texts that satisfy the constraint. for the sake of simplicity, we will denote without distinction the class, the constraint verified by its texts and the associated control code by c. trained with different control codes, the model learns p θ (x 1:t | c) = t @xmath0:t-1 , c). the constraint can then be applied during generation by appending the corresponding control code to the prompt. while this method gives some kind of control over the generation, the control codes need to be defined upfront and the lm still needs to be trained specifically for each set of control codes. this is an important limitation since the current trend in text generation is the use of large pre-trained models which can hardly be fine-tuned (for instance, the last version of gpt, gpt-3, cannot be fine-tuned without access to very large hardware resources). ## ppl-mcts method the approach that we propose is in line with methods using a discriminator to guide a large lm decoding, without the need to re-train it. also, it can be applied to any lm with any discriminator, following the plug and play paradigm. unlike previous approaches, it is able to have a long term vision on what is generated. being able to make a short-term decision (choice of the next token x t at time step t) that is promising in the long run is based on the exploration of the search space. we propose here to use the monte carlo tree search (mcts) for an efficient exploration of this space. mcts is very well suited for this problem for three reasons. first, it allows to get a local score (i.e, a score for the next token to emit) using finished sequences. hence, this score is more meaningful than scores based only on the next step. second, it allows to explicitly define the compromise between exploitation of promising sequences (with a high likelihood), and exploration of other potentially promising sequences (to not miss better sequences with a lower likelihood). the fact that regret, i.e the number of simulations done on a suboptimal sequence, has a theoretical upper bound in mcts (rosin, 2011) is a nice guarantee that the computation time is not wasted and the search is efficient. finally, it outputs a solution at each iteration and so can fit our computational budget by allowing to adjust the quality of the solution to calculation spent. ## conclusion in this paper, we show that it is possible to control generation with the help of a discriminator that implements some expected constraints on the text during decoding. this flexible approach is very useful when using very large language models, such as gpt-3, whose fine-tuning computational costs are prohibitive. in contrast, training a discriminator is easier and cheaper. our proposed methods, that mix the discriminator constraint and the generation, yield performance that is equivalent to the best approaches based on lm tuning at lower training cost. on the other hand, such approaches have an additional cost during inference because of the cost of the discriminator being applied to candidate generations. a study on this additional cost depending on the type of discriminator used can be found in @xcite . ppl-mcts offers a solution for cases where training is too costly for the downstream application or the language model is not directly accessible. seeing text generation as a tree exploration process, an existing approach such as gedi indeed lowers the cost of width exploration but the depth exploration is still an issue. using gedi for constrained generation is thus very similar to a standard maximum likelihood search which still lacks of an optimal search method. on the other hand, monte carlo tree search provides an efficient way to explore the tree by determining the best local choice in the long run, lowering the cost of depth exploration. thus, these two methods solve different facets of constrained generation, and the combination of the two is a promising perspective. moreover, mcts allows to precisely define the best compromise between cost and quality through the number of iterations and the roll-out size, while ensuring the efficiency of the search theoretically. for reproducibility purposes, our implementation is made available at https: //github.com/nohtow/ppl-mcts . several research avenues are opened by this work. for methods yielding high perplexity, it would be interesting to explore how to set the α parameter in order to reach the best compromise between accuracy and perplexity. similarly, the size (number of tokens considered) of the rollout in mcts offers some ways to control the cost/performance compromise. an adaptive rollout size, for example rolling-out until the score of the discriminator is above or below a threshold as in @xcite , would seem particularly suited for texts. it should also be noted that finetuning a model and controlling the generation with a discriminator can be used jointly. for instance, one can use ppl-mcts on a tuned lm, which will most likely result in even better performances because sequences considered during the search will have an overall higher quality for the considered task. finally, not only can ppl-mcts be applied to any property that a discriminator can identify, but it can also work using other scoring methods (human evaluation, regular expressions, heuristic based evaluation, ...) as long as the score reflects compliance with the expected property.