Who Needs Decoders? Efficient Estimation of Sequence-Level Attributes with Proxies (2024)

Abstract: Sequence-to-sequence models often require an expensive autoregressive decoding process. However, for some downstream tasks such as out-of-distribution (OOD) detection and resource allocation, the actual decoding output is not needed, just a scalar attribute of this sequence. In such scenarios, where knowing the quality of a system’s output to predict poor performance prevails over knowing the output itself, is it possible to bypass the autoregressive decoding? We propose Non-Autoregressive Proxy (NAP) models that can efficiently predict scalar-valued sequence-level attributes. Importantly, NAPs predict these metrics directly from the encodings, avoiding the expensive decoding stage. We consider two sequence tasks: Machine Translation (MT) and Automatic Speech Recognition (ASR). In OOD for MT, NAPs outperform ensembles while being significantly faster. NAPs are also proven capable of predicting metrics such as BERTScore (MT) or word error rate (ASR). For downstream tasks, such as data filtering and resource optimization, NAPs generate performance predictions that outperform predictive uncertainty while being highly inference efficient.

## introduction autoregressive models have emerged as the dominant approach for many sequence-to-sequence tasks @xcite @xcite @xcite and are the state-of-theart for a range of tasks such as automatic speech recognition (asr) @xcite , machine translation (mt) @xcite , and abstractive text summarization @xcite . however, for many applications, the decoded output sequence is not required, only attributes of the sequence. in out-of-distribution (ood) detection, only a sequence-level metric such as confidence is required @xcite . in selective classification @xcite @xcite the output is only needed if the prediction is trusted. another example is deferral strategies for resource allocation @xcite @xcite @xcite , where computation is allocated between systems of different complexity. standard deferral strategy approaches use the predictive uncertainty of a simpler system to decide whether or not to pass it on to a better-performing system of higher complexity @xcite . all of the examples above require some form of predictive uncertainty metric from the output, which in the case of transformer-based autoregressive models are expensive to obtain @xcite @xcite @xcite . combined with the quadratic cost of self-attention @xcite and autoregressive decoding (equipped with beam-search @xcite ), this can limit the application of these systems in real-world settings, such as those that have limited computational resources or require low latency @xcite . furthermore, ensembling generally improves system performance and can be leveraged for useful analysis, such as for robust uncertainty estimation @xcite . however, ensembles' memory and inference costs scale linearly with the number of members in the ensemble, making them even more impractical for real-world scenarios. there are methods including knowledge distillation (kd) @xcite and ensemble distribution distillation (edd) @xcite @xcite that distill knowledge from an autoregressive ensemble but this does not circumvent the high costs fundamen-tally associated with autoregressive generation. previous works have investigated adding a second output head explicitly trained to capture a specific metric such as epistemic uncertainty in image segmentation @xcite or the true class probability in image classification @xcite . the work of @xcite extends this style of approach to asr by adding a second head to the decoder, to predict token-level decoding errors. despite its success in providing robust estimates, computing the output uncertainties still requires an expensive autoregressive decoding process. the work of @xcite trains an independent proxy model for estimating uncertainties. this method is based on training a much smaller image classification model in an identical manner to the primary model, instead using the uncertainties produced by the small model's outputs to guide the primary one. in the space of autoregressive encoder-decoder models, this approach is still not feasible; the costs of training and decoding persist even for small autoregressive models. in this paper, we propose non-autoregressive proxy (nap) models that directly estimate sequence-level attributes, bypassing the expensive autoregressive decoding process. when deployed, these lightweight proxy models can be used to robustly predict sequence properties using a fraction of the computational requirements. our approach is kept general and applicable to any sequence attribute, demonstrating the usefulness of this framework to diverse metrics such as sequencelevel predictive uncertainty, bertscore for mt, and word error rate (wer) for asr. investigations into downstream tasks such as out-of-distribution (ood) detection show that naps can outperform an ensemble at a fraction of the inference time. due to the flexibility of the proposed framework, we also investigate training naps on sequence-level performance metrics (bertscores and wers), outperforming uncertainty-based approaches on data filtering and resource optimization. ## background there has been a range of work on predicting sequence-level attributes. one common example is estimating uncertainties from the outputs of autoregressive systems @xcite , where unsupervised token-level uncertainties from some decoding process are combined to form sequence-level estimates. such sequence-level uncertainties are then used in downstream tasks such as ood detection @xcite , quality estimation @xcite and curriculum learning @xcite . previous work has also explored task-specific supervised approaches to confidence/metric estimation. the work of @xcite ) explores training a small independent model to predict the sub-utterance-level word error rate (wer) of a primary asr model for short-duration audio when the reverberant conditions change. however, the approach is not generalizable to other domains such as mt due to the specific focus on reverberant speech. other work has also focused on training an error detection module attached to the decoder of some asr or mt system @xcite @xcite @xcite . for example, a typical approach to training the decoder-side error detector is based on token-level error labels from the minimum levenshtein distance alignment to the ground truth. from these token-level estimates, a sequence-level confidence score can be derived. in asr where there is often one clear true transcription of the input audio, such an error detection module is appropriate. however, these approaches are inappropriate for mt where multiple translations could all have the same meaning and be considered valid. such a token-level error detector would flag other valid translations as errorful even when conveying the same information and meaning. this final example is one of the main motivations behind bertscore and related approaches @xcite @xcite . bleu @xcite has long been the main mt evaluation metric for measuring sequence similarity between a translation and a reference using some measure of overlap. however, it suffers from similar issues as (levenshtein) edit-distance metrics. bertscore resolves such issues by leveraging bidirectional language models in generating contextual variable-length embeddings for both the translation and reference sequence, computing an automatic sequence similarity score in this embedding space. there has also been a set of work on supervised mt quality estimation @xcite @xcite in which models are trained to estimate the quality (human expert estimated metric) of a translation by making use of the source, the decoded translation and additional token-level prob-ability. however, both the automatic bertscore and quality metrics require an expensive autoregressive decoding stage to obtain the estimate. ## non-autoregressive proxy we are interested in the general problem of estimating sequence-level attributes whilst remaining highly inference-efficient. these sequencelevel metrics include: (1) information-theoretic uncertainties @xcite ; (2) neuralbased evaluation scores such as bertscore @xcite ; and (3) discrete sequence-similarity metrics such as word error rate. the standard approach to obtaining these sequence-level metrics is to run an expensive autoregressive decoding scheme to produce a set of hypotheses. one can either extract sequence attributes directly from this hypothesis set @xcite or compare them with their corresponding references to obtain a measure of sequence similarity. the aim of this paper is to avoid the costly autoregressive generation stage and instead train an encoder-only, non-autoregressive proxy (nap) model to imitate the sequence metrics produced by an autoregressive system, using only the source, see figure 1 . we employ two different setups as shown in figures 1a and 1b. the aim of the first setup is to train a proxy to directly extract sequence uncertainties when the main model is additionally given the reference sequence. this is in order to teach the proxy model to imitate the uncertainties from the gold reference. the second setup aims to teach the proxy a sequence similarity score when the autoregressive generated hypothesis is compared to the reference. both setups are highly challenging as the non-autoregressive proxy is tasked with predicting sequence-level metrics from only the source. however, the key feature of the nap is that it directly predicts these metrics without a decoding scheme (e.g. beam search) and without any reference sequences, allowing the user to extract useful information from large amounts of unlabelled data with little cost. furthermore, in the first setup of figure 1a , the proxy also avoids the exposure bias problem @xcite , by directly training on the teacher-forced @xcite ) sequence uncertainties. in this work, we follow figure 1a in training a proxy on both single teacher confidence and entropy scores or ensemble mutual information, evaluating its imitation ability and downstream out-of-distribution detection ability. we also follow figure 1b in training a proxy to predict bertscores in machine translation and wer in speech recognition and evaluate the performance of the nap on a data filtering and resource optimization task. loss function: sequence-level metrics are represented by single scalar values. therefore, the proxy student can be trained using any regression loss function. however, unlike standard regression tasks, we seek to learn the relative ordering (rankings) of our scores, as this simplifies the task and is more pertinent for downstream applications such as ood detection. therefore, we will mainly opt for the spearman rank and pearson correlation coefficient (scc & pcc) depending on the specific task considered. consider a batch of n items with teacher scores {s i } n @xmath0. the spearman loss function is then defined as: predictor design: in order to produce a scalar score from a variable-length encoder-output representation, we make use of a pooling operation. we utilize two options, temporal averaging or multihead attention with a single trainable query. the encoder vector outputs {v l } l @xmath1. furthermore, early exploratory experiments found that a softmax activation is vital for good performance as it can be seen as introducing inductive bias into the estimation of information-theoretic and related metrics. details on mlp architecture and ablation studies are provided in appendix b.2. proxy encoder backbone: by default, the nap backbone is initialized from the encoder weights proxy encoder predictor source encoder decoder source reference sequence uncertainty output loss (a) setup 1: capturing sequence uncertainties. proxy encoder predictor source encoder decoder source reference sequence similarity hypothesis loss (b) setup 2: capturing sequence similarities. of the main encoder-decoder model. since pretrained models such as t5 @xcite and whisper @xcite are released in different sizes, one can utilize smaller architectures to initialize smaller proxies, and train them to predict attributes of larger systems. appendix b.4 further explores 'mismatched' encoders, e.g. using a roberta nap to predict the output attributes of a t5 system. furthermore, all experiments in this paper freeze the encoder backbone and only train the small predictor on top of the nap encoder. this improves the training speed and memory usage allowing a user to train multiple predictor heads on top of the same backbone, each for a different metric (e.g. estimating sequence-level confidence and bertscores in the same forward pass). note that the purpose of our investigations is not to create the best possible nap model (for example, finetuning the backbone encoder could improve performance at no cost of inference speed). we only seek to demonstrate that this approach is highly flexible and applicable to a range of sequence-level metrics and can provide cheap but useful information for sequence-to-sequence tasks. ## experimental evaluation predicting uncertainties: we will evaluate the imitation ability of nap models on various tasks. following setup 1, the first set of experiments will focus on the ability of a proxy system to capture sequence-level confidence or entropy from a single t5 transformer @xcite finetuned on a spoken-language machine translation (mt) dataset. we further explore the ability of naps to imitate mutual information (epistemic uncertainty @xcite ) from an ensemble of t5 systems. the performance of the naps will then be evaluated by measuring the spearman rank correlation between the teacher (under teacher-forcing @xcite ) and the proxy estimates on a range of in-domain (id) and out-of-domain (ood) datasets. we also investigate the performance of the proposed nap on ood detection. predicting bertscores: following setup 2, we also investigate if proxy systems can capture much more complex sequence metrics such as bertscores @xcite ) from a single t5 in mt. capturing this metric is especially challenging since the beam-search output of the t5 decoder and corresponding reference will be fed through a language model such as bert @xcite which then computes the final score. the performance will be measured by computing the spearman rank between proxy outputs and bertscores on both id and ood datasets. furthermore, the proxy is compared to sequence-level confidence and entropy scores from the t5 model to see how well they correlate with bertscores. the performance of a bertscore estimating proxy system can also be evaluated on two downstream tasks: filtering task @xcite : given a dataset, we remove the examples with the lowest proxy or highest uncertainty estimate. for good estimates, the filtered subset should display a higher average bertscore. resource optimization task @xcite : under a fixed resource budget, one seeks to allocate inputs to models of different complexity in order to maximize performance. a well-performing allocation system would achieve higher performance with a smaller budget, see figure 2 . small model source large model hypothesis hypothesis if high uncertainty (a) baseline deferral system. if high score proxy model source small model score hypothesis if low score large model hypothesis (b) proxy deferral system. predicting wer: finally, we follow setup 2 in investigating if a nap can imitate the sentencelevel wer and the total number of errors produced by an asr system. in this case, we utilize the pretrained state-of-the-art whisper @xcite models on the librispeech corpus @xcite . since the whisper model is very well-performing, it is able to perfectly decode a large fraction of the dataset, which would cause issues for a rank-based loss such as spearman. we, therefore, resort to pearson for these experiments. note, the corpus-level wer performance of an asr system is a length-weighted average of the sentence-level wers. therefore, we also train naps to predict the number of decoding errors in an utterance. similar to the bertscore experiments, the performance of naps will be evaluated in a similar manner using both filtering and resource optimization tasks. ## conclusion for many downstream sequence-to-sequence tasks, only attributes of the output sequence are needed, and not the output itself. in this paper, we propose a simple efficient framework for directly estimating scalar sequence-level attributes using only the source. while conditioning on the decoding can provide performance gains, this fundamentally defeats the idea behind the inference-efficient nonautoregressive proxies which make them useful and practical for preemptive performance prediction. we show that naps can learn informationtheoretic uncertainties as well as performance metrics, such as bertscores for mt or wers for asr, in terms of both mimicking attribute score ranks and the impact on downstream tasks. for mt systems they outperform a deep ensemble on ood detection with an order of magnitude higher inference speed. furthermore, naps are able to outperform predictive uncertainty on downstream tasks such as data filtering and resource optimization on both asr and mt tasks.