Inference-Time Policy Adapters ( IPA ): Tailoring Extreme-Scale LM s without Fine-tuning (2023)

Abstract: While extreme-scale language models have demonstrated exceptional performance on a variety of language tasks, the degree of control over these language models through pure prompting can often be limited. Directly fine-tuning such language models can be effective for tailoring them, but it can be either extremely costly (e.g., GPT-3) or not even feasible for the broader community (e.g., GPT-4). We propose Inference-time Policy Adapters (IPA), which efficiently tailors a language model such as GPT-3 without fine-tuning it. IPA guides a large base model during decoding time through a lightweight policy adapter trained to optimize an arbitrary user objective with reinforcement learning. On five challenging text generation tasks, such as toxicity reduction and lexically constrained generation, IPA consistently brings significant improvements over off-the-shelf language models. It outperforms competitive baseline methods, sometimes even including expensive fine-tuning. In particular, tailoring GPT-2 with IPA can outperform GPT-3, while tailoring GPT-3 with IPA brings a major performance boost over GPT-3 (and sometimes even over GPT-4). Our promising results highlight the potential of IPA as a lightweight alternative to tailoring extreme-scale language models.

## introduction large language models (llms) have recently shown remarkable progress in various text generation tasks by adapting to instructions or examples @xcite . however, the degree of control (e.g., the inclusion of keywords, avoiding harmful language) offered by these extreme-scale models through pure prompting is still limited @xcite . moreover, prompting can be a brittle process due to llms being overly sensitive to the surface-form of the instructions @xcite . furthermore, even with a carefully written prompt, llms may still struggle to fulfill certain task requirements due to their inherent limitations @xcite . resource-intensive fine-tuning, through supervised learning, and more recently reinforcement learning (rl) @xcite have shown promise in tailoring language models to arbitrary user-given objectives. rl, in particular, known for its generalizability and flexibility, allows models to learn from desired rewards. however, these methods require accessing and updating models parameters, which can be extremely large or inaccessible in state-of-the-art models like @xcite . this limitation makes fine-tuning unfeasible for the broader community. alternatively, inference-time algorithms can tailor a language model without accessing its parameters. these algorithms align language models' outputs with desired task/user-specific properties by adjusting the model's output distribution based on certain task-specific heuristics, while leaving the underlying model untouched. despite the progress, these approaches are either restricted to specific tasks @xcite , require domain-specific knowledge @xcite , suffer from expensive run-time at inference @xcite @xcite , or have shown to be less effective compared to direct rl optimization @xcite . drawing inspiration from rl and inferencetime techniques, we propose inference-time policy adapters ( ipa), an efficient and generalizable algorithm, which tailors a large language model at decoding-time toward desired objectives without fine-tuning it. to do so, ipa combines a large base lm's output distribution with that of a smaller-sized model (a lightweight adapter policy), and optimizes the combined distribution towards a given objective with rl (figure 1 ). ipa uses two key ideas to make learning efficient. first, ipa only updates the adapter's parameters, avoiding the need to update the large base lm. second, ipa replaces the large base model with an approximate policy-a smaller model that approximates the base model's distribution. the approximate policy is either a smaller model from the same language model family or a distilled version of the base model. at inference time, we decode with the combined distribution of the base model and the trained policy adapter. experiments across five challenging text generation tasks show that ipa brings consistent improvements over off-the-shelf language models, outperforming competitive baselines -sometimes even including expensive fine-tuning. in particular, tailoring gpt-2 with ipa can outperform gpt-3, while tailoring gpt-3 with ipa brings a major performance boost over gpt-3 (and sometimes even over gpt-4). our compelling highlight the promise of ipa as a lightweight alternative for tailoring large language models to a wide range of objectives. ipa opens new ways to augment or customize extreme-scale language models using only academic-level resources. ## background in this section, we introduce our text generation setting ( §2.1) and a brief background on tailoring language models with reinforcement learning ( §2.2). we then introduce our ipa algorithm for tailoring large language models without fine-tuning ( §3). ## inference-time policy adapters (ipa) we introduce inference-time policy adapters (ipa), a lightweight approach to tailor language models towards a user-specified objective. ipa trains a small adapter policy that adjusts the outputs of a (larger) base model at inference-time in order to maximize a reward. in doing so, ipa avoids the cost of updating the large base model, without the need to hand-design inference-time heuristics. ## experiments we evaluate ipa on a diverse range of tasks: toxicity reduction ( §4.1), lexically constrained generation ( §4.