Large language models ( LM ) based on Transformers allow to generate plausible long texts . In this paper , we explore how this generation can be further controlled at decoding time to satisfy certain constraints ( e.g. being nontoxic , conveying certain emotions , using a specific writing style , etc . ) without fine-tuning the LM . Precisely , we formalize constrained generation as a tree exploration process guided by a discriminator that indicates how well the associated sequence respects the constraint . This approach , in addition to being easier and cheaper to train than fine-tuning the LM , allows to apply the constraint more finely and dynamically . We propose several original methods to search this generation tree , notably the Monte Carlo Tree Search ( MCTS ) which provides theoretical guarantees on the search efficiency , but also simpler methods based on reranking a pool of diverse sequences using the discriminator scores . These methods are evaluated , with automatic and human-based metrics , on two types of constraints and languages : review polarity and emotion control in French and English . We show that discriminatorguided MCTS decoding achieves state-of-theart results without having to tune the language model , in both tasks and languages . We also demonstrate that other proposed decoding methods based on re-ranking can be really effective when diversity among the generated propositions is encouraged . Generative language models exist for a long time , but with advent of the transformer architecture ( Vaswani et al . , 2017 ) and increasing computing capabilities , they are now able to generate well written and long texts . In particular , large models , such as the well known GPT-2 ( Radford et al . , 2019 ) and GPT-3 ( Brown et al . , 2020 ) , have been used successfully for various applications : assisting writers , summarizing , augmentating data for subsequent NLP tasks , generating fake news ( Kumar et al . , 2020 ; Papanikolaou and Pierleoni , 2020 ; Zellers et al . , 2019 ) . Yet , beside the prompt used to initiate the generation process , there are few options to have control on the generation process . Being able to add some constraints on the generated texts is useful for various situations . For example , it allows to create texts that follow a certain writing style , convey a certain emotion or polarity or to ensure that a generated summary contains correct information . More critically , it can be used to prevent the inherent toxicity of language models trained on the internet , or to not reproduce gender or race stereotypes . So far , most methods necessitate to fine-tune the LM , so that it specifically learns to model this constraint , i.e. the constraint is -hopefully-incorporated in the LM . This finetuning approach has several drawbacks . It implies to train multiple specific LMs ( one per constraint ) , which is costly , when even possible given the size of current state-of-the-art LM , and results in several models . In this paper , we propose new approaches to add such additional constraints on the texts but at decoding time . We exploit a discriminator that is trained to determine if a text follows a given constraint or not ; its output provides information to guide the generation toward texts that satisfy this expected constraint . In order to make the most of the discriminator information , we propose an original method based on the Monte Carlo Tree Search ( MCTS ) algorithm ( Coulom , 2006 ) , namely Plug and Play Language -Monte Carlo Tree Search ( PPL-MCTS ) . We also propose simpler methods based on re-ranking to fulfil this goal . Both approaches do not require to fine-tune the LM ; adding a new constraint can thus simply be done by providing a discriminator verifying if a text complies with what is expected . More precisely , our main contributions are the following ones : 1 . we propose to use MCTS as a decoding strat-egy to implement constrained generation and we show , on 3 datasets and 2 languages , that it yields state-of-the-art results while offering more flexibility ; 2 . we also explore simpler generation methods based on re-ranking and show that this kind of approach , with low computational costs , can also be competitive if the diversity within propositions to re-rank is encouraged ; 3 . we provide a fully functional code implementing a batched textual MCTS 1 working with the popular HuggingFace 's Transformers library ( Wolf et al . , 2020 ) 2 Related work The goal of constrained textual generation is to find the sequence of tokens x 1 : T which maximises p(x 1 : T | c ) , given a constraint c. Few methods address the constrained textual generation . Class-conditional language models . Classconditional language models ( CC-LMs ) , as the Conditional Transformer Language ( CTRL ) model ( Keskar et al . , 2019 ) , train or fine-tune the weights θ of a single neural model directly for controllable generation , by appending a control code in the beginning of a training sequence . The control code indicates the constraint to verify and is related to a class containing texts that satisfy the constraint . For the sake of simplicity , we will denote without distinction the class , the constraint verified by its texts and the associated control code by c. Trained with different control codes , the model learns p θ ( x 1 : T | c ) = T t=1 p θ ( x t | x 1 : t-1 , c ) . The constraint can then be applied during generation by appending the corresponding control code to the prompt . While this method gives some kind of control over the generation , the control codes need to be defined upfront and the LM still needs to be trained specifically for each set of control codes . This is an important limitation since the current trend in text generation is the use of large pre-trained models which can hardly be fine-tuned ( for instance , the last version of GPT , GPT-3 , can not be fine-tuned without access to very large hardware resources ) . Discriminator-based methods The general idea of discriminator-guided generation is to combine 1 https://github.com / NohTow / PPL-MCTS a disciminator D with a generative LM . The discriminator explicitly models the constraint by calculating the probability p D ( c | x 1 : T ) of the sequence x 1 : T to satisfy the constraint c. This probability is directly related to p(x 1 : T | c ) through Bayes ' rule : p(x 1 : T | c ) ∝ p D ( c | x 1 : T ) p θ ( x 1 : T ) . Discriminator-based methods alleviate the training cost problem , as discriminators are easier to train than a LM . Moreover , any additional constraint can be defined a posteriori without tuning the LM , only by training another discriminator . The discriminators have been used in different ways to explore the search space . In the work of ( Holtzman et al . , 2018 ; Scialom et al . , 2020 ) , the space is first searched using beam search to generate a pool of proposals with a high likelihood p θ ( x 1 : T ) , and then the discriminator is used to re-rank them . However , in addition that beam search can miss sequences with high likelihood , it is biased towards the likelihood , while the best sequence might only have an average likelihood , but satisfies the constraint perfectly . Hence , it might be more suitable to take the discriminator probability into account during decoding rather than after generating a whole sequence . In this case , the discriminator is used at each generation step t to get the probability p D ( c | x 1 : t ) for each token of the vocabulary V , and merge it to the likelihood p θ ( x 1 : t ) to choose which token to emit . In order to reduce the cost of using a discriminator on every possible continuation , GeDi ( Krause et al . , 2020 ) proposes to use CC-LMs as generative discriminators . The method relies on the fact that the CC-LM computes p θ ( x t | x 1 : t-1 , c ) for all tokens of the vocabulary which can be used to get p θ ( c | x 1 : t ) for all tokens using Bayes ' equation . This approach is thus at the intersection of tuning the LM and using a discriminator : it tunes a small LM ( the CC-LM ) to guide a bigger one . In Plug And Play Language Model ( PPLM ) ( Dathathri et al . , 2020 ) , the discriminator is used to shift the hidden states of the pre-trained transformer-based LM towards the desired class at every generation step . PPLM can be used on any LM and with any discriminator . However , PPLM needs to access the LM to modify its hidden states , while our approach only requires the output logits . As some LM can only be used through access to logits ( e.g. GPT-3 API ) , this makes our approach more plug and play than PPLM . A common drawback of all these approaches is their lack of a long-term vision of the generation . Indeed , the discriminator probabilities become necessarily more meaningful as the sequence grows and might only be trustable to guide the search when the sequence is ( nearly ) finished . When used in a myopic decoding strategy , classification errors will cause the generation process to deviate further and further . Trying to optimize a score defined in the long horizon by making short term decisions is very similar to common game setups such as chess , where the Monte Carlo Tree Search ( MCTS ) has proven to be really effective ( Silver et al . , 2018 ) , which motivated our approach . In this paper , we show that it is possible to control generation with the help of a discriminator that implements some expected constraints on the text during decoding . This flexible approach is very useful when using very large language models , such as GPT-3 , whose fine-tuning computational costs are prohibitive . In contrast , training a discriminator is easier and cheaper . Our proposed methods , that mix the discriminator constraint and the generation , yield performance that is equivalent to the best approaches based on LM tuning at lower training cost . On the other hand , such approaches have an additional cost during inference because of the cost of the discriminator being applied to candidate generations . A study on this additional cost depending on the type of discriminator used can be found in ( Chaffin et al . , 2022 ) . PPL-MCTS offers a solution for cases where training is too costly for the downstream application or the language model is not directly accessible . Seeing text generation as a tree exploration process , an existing approach such as GeDi indeed lowers the cost of width exploration but the depth exploration is still an issue . Using GeDi for constrained generation is thus very similar to a standard maximum likelihood search which still lacks of an optimal search method . On the other hand , Monte Carlo Tree Search provides an efficient way to explore the tree by determining the best local choice in the long run , lowering the cost of depth exploration . Thus , these two methods solve different facets of constrained generation , and the combination of the two is a promising perspective . Moreover , MCTS allows to precisely define the best compromise between cost and quality through the number of iterations and the roll-out size , while ensuring the efficiency of the search theoretically . For reproducibility purposes , our implementation is made available at https : //github.com / NohTow / PPL-MCTS . Several research avenues are opened by this work . For methods yielding high perplexity , it would be interesting to explore how to set the α parameter in order to reach the best compromise between accuracy and perplexity . Similarly , the size ( number of tokens considered ) of the rollout in MCTS offers some ways to control the cost / performance compromise . An adaptive rollout size , for example rolling-out until the score of the discriminator is above or below a threshold as in ( Cotarelo et al . , 2021 ) , would seem particularly suited for texts . It should also be noted that finetuning a model and controlling the generation with a discriminator can be used jointly . For instance , one can use PPL-MCTS on a tuned LM , which will most likely result in even better performances because sequences considered during the search will have an overall higher quality for the considered task . Finally , not only can PPL-MCTS be applied to any property that a discriminator can identify , but it can also work using other scoring methods ( human evaluation , regular expressions , heuristic based evaluation , ... ) as long as the score reflects compliance with the expected property .

Challenge: Finetuning different models with different constraints and controling text generation models to avoid outputting toxic outputs is computationally expensive.
Approach: They propose to use a discriminative model during generation time without fine-tuning using Monte Carlo Tree Search for efficiency.
Outcome: The models with guidance at inference time using a discriminator outperform the best models in French and English on automatic and manual metrics.