Efficient Sample-Specific Encoder Perturbations (2024)

Abstract: Encoder-decoder foundation models have displayed state-of-the-art performance on a range of autoregressive sequence tasks. This paper proposes a simple and lightweight modification to such systems to control the behaviour according to a specific attribute of interest. This paper proposes a novel inference-efficient approach to modifying the behaviour of an encoder-decoder system according to a specific attribute of interest. Specifically, we show that a small proxy network can be used to find a sample-by-sample perturbation of the encoder output of a frozen foundation model to trigger the decoder to generate improved decodings. This work explores a specific realization of this framework focused on improving the COMET performance of Flan-T5 on Machine Translation and the WER of Whisper foundation models on Speech Recognition. Results display consistent improvements in performance evaluated through COMET and WER respectively. Furthermore, experiments also show that the proxies are robust to the exact nature of the data used to train them and can extend to other domains.

## introduction encoder-decoder models have displayed state-ofthe-art performance in a wide range of sequence tasks @xcite including machine translation (mt) @xcite , abstractive text summarization & question answering @xcite and automatic speech recognition (asr) @xcite . however, the standard approach to training these systems often relies on teacher-forcing with the likelihood criteria, e.g. next token prediction of the reference sequence. while this framework has been shown successful and reliable in the tasks above, often the desired criteria are some sequence-level non-differentiable performance measures. in mt the desired criteria was the n-gram-based (sacre)bleu @xcite , overtaken more recently by the neural-based comet @xcite evaluation metric. in asr the criteria is the word error rate (wer) measuring the rate of substitutions, insertions, and deletions between the decoded output and the reference. prior work has addressed the exposure bias arising from training in teacher-forcing @xcite and the loss mismatch between the likelihood and the desired sequence-level loss @xcite @xcite @xcite @xcite @xcite . both of these approaches modify the training so it more closely links with how the model would be used during deployment. however, in the regime of large pre-trained foundation models, re-training such systems is computationally expensive and unstable. other approaches attack this problem at inference time by merging outputs from several systems guided by appropriate sequence-level metrics @xcite @xcite @xcite . whilst these approaches have shown promising gains, they rely on the use of ensembles which have significantly higher computational costs. furthermore, they are not generalizable to any metric. in this paper, we propose a novel, simple and efficient approach to modify the behaviour of a single frozen pre-trained encoder-decoder foundation model. we show that it is possible to perturb the outputs of the encoder to trigger the decoder to produce better-performing decodings according to some pre-selected generic attribute. unlike prior approaches, our novel proposal applies to frozen pre-trained systems and only leads to an insignificant increase in runtime. this paper is focused on showing the efficacy of the approach in improving the comet performance of flan-t5 @xcite on nmt and the wer performance of whisper @xcite on asr. furthermore, the approach is generalizable and applicable to other attributes such as the sentiment of outputs. ## background reinforcement learning (rl) and minimum bayes risk (mbr) decoding have been two paradigms used to align a sequence model and minimize the impact of exposure bias. the former is traditionally used to improve the training while the latter is used to modify the decoding procedure. both come with their own sets of advantages and disadvantages. while often leading to better performance, these approaches are often expensive to use during training and/or inference. in the works of bahdanau et al. (2017); lamb et al. (2016) , a second network is used, either as a critic in an actor-critic framework @xcite or as a discriminator in a generative adversarial framework @xcite . the aim of both of these networks, although mechanically different, is to ensure the training procedure resembles the inference stage, minimising the effect of exposure bias. on the other hand, the work of @xcite explored a post-training approach in which the minimum bayes risk decoder samples many sequences from the system and chooses the sample with the lowest risk. however, the efficacy of this system is highly dependent on being able to sample a large set of outputs, a feat not possible for large pre-trained systems. alternatively, @xcite applies this approach to an ensemble of similarly performing systems, without regard to the inference cost. there is also a body of work on memory and inference-efficient adaptation of foundation models such as prefix-tuning (li and liang, 2021) and low-rank adaptation @xcite . these approaches are effective at adapting a foundation model to a certain task at the cost of degrading other abilities. in addition, these approaches still require back-propagating through the whole foundation model making them potentially expensive to train and mainly target teacher-forcing likelihood training. finally, the work of @xcite introduced a general framework in which an encoder is extended with a small non-autoregressive proxy (nap) trained to directly capture an arbitrary metric. while only applicable to encoder-decoder systems, it showed that estimates produced by nap systems were useful in downstream tasks. ## perturbations of encoders the proposal's core is a flexible and efficient approach for augmenting the behaviour of an en-coder on a sample-by-sample basis to trigger a better decoder performance. the starting point for such a goal will be the recently introduced non-autoregressive proxy (nap). while the original work trained the network on a specific metric and used the estimates at runtime to directly perform various downstream tasks, we will extend the view of this network to a differentiable approximation of a sequence-level metric, and use the gradients of this approximation to improve performance. let ϕ e and ϕ d represent the parameters of some encoder and decoder network, and let x be some input (token or embedding) sequence at inference time we have e 1:@xmath0; ϕ e ) where e 1:l represents a sequence of l encoder embeddings that are consumed by the decoder. the decoder, through an autoregressive process, produces an output sequence ŷ = f (e 1:l ; ϕ d ). we aim to find a samplespecific perturbation δ 1:l to the sequence of encoder outputs such that ȳ = f (e 1:l + δ 1:l ; ϕ d ) gives us a higher score according to some score s (e.g. comet): ## experimental evaluation the first set of experiments aim to evaluate the efficacy of the gradient perturbation approach on nmt using the pre-trained flan-t5. we resort to the opus-100 @xcite dataset and use two splits, the english to german (@xmath1, each with a training set of 1m sentence pairs. for each, we use the flan-t5 system to decode the data using greedy search with a maximum of 128 output tokens. the decodings are scored using comet (unbabel/wmt22-comet-da) @xcite . next, we take these naps and use the gradient with respect to the encoder outputs to perform the augmentation detailed in equation ( 2 ), see figure 1 . this shows that the gradients derived from a lightweight nap trained on comet scores can be used to obtain some performance gains. while small steps α barely have an impact, large α lead to a degradation in performance showcasing the importance of finding a good α. furthermore, we find that although the naps were trained on the comet scores of greedy decodings, they can still be used to improve beam search. we also observe that naps trained in a certain translation direction can still be used for other directions. for example, nap en-es is still able to improve en-de performance. we also investigate this for a range of flan-t5 models, see ## conclusion this paper presents a lightweight modification of frozen (pre-trained) encoder-decoder systems by training a small network to predict a metric and use this as a differentiable extension. the gradients of this small proxy network have been shown useful in improving the performance on a range of sequence tasks such as the flan-t5 comet performance on nmt and whisper wer performance on asr.