Multi-Source Test-Time Adaptation as Dueling Bandits for Extractive Question Answering (2023)

Abstract: In this work, we study multi-source test-time model adaptation from user feedback, where K distinct models are established for adaptation. To allow efficient adaptation, we cast the problem as a stochastic decision-making process, aiming to determine the best adapted model after adaptation. We discuss two frameworks: multi-armed bandit learning and multi-armed dueling bandits. Compared to multi-armed bandit learning, the dueling framework allows pairwise collaboration among K models, which is solved by a novel method named Co-UCB proposed in this work. Experiments on six datasets of extractive question answering (QA) show that the dueling framework using Co-UCB is more effective than other strong baselines for our studied problem.

## introduction large language models (llms) can be fine-tuned or prompted with texts to achieve good performance in nlp tasks @xcite @xcite . however, because of the unexpected distribution shift at test time, the effectiveness of llms can degenerate @xcite . they may also generate outputs that are untrustworthy or toxic and fail to meet user expectations @xcite . one critical issue that we need to address is to improve the generalization ability of llms. recent research on test-time adaptation (tta) suggests a possible way to do this, by continually updating the deployed model with target data from an arbitrary test distribution @xcite . interacting with users is important during testtime adaptation. first, user feedback allows the model to better align with humans @xcite . users can directly teach the model to learn by interaction so that the 1 code of the paper is available at https://github.com/ oceanypt/multi-source-tta . model can be better trained to follow human instructions and reduce the generation of toxic and harmful content. besides, obtaining feedback from users can also reduce the cost of data annotation by experts, and the collected data will be more in line with the distribution of the users @xcite , which makes the adaptation more economical and effective. leveraging multiple learned models of tasks is also important for tta. as in previous work, utilizing multiple known tasks helps the model better learn new tasks (or distributions), such as metalearning @xcite and multi-source domain adaptation @xcite . to take advantage of known tasks, compared to reusing task data, directly using their learned models has gained popularity recently @xcite , which is much cheaper for online adaptation and has better data privacy protection @xcite . recent work on lightweight tuning empowers llms to store knowledge of a large number of tasks cheaply @xcite . platforms like huggingface @xcite also allow users to share locally trained models, promoting a large amount of knowledge stored as models in the cloud. so, it has become more critical for tta to adapt from multiple learned models of tasks. based on the above discussion, we propose to study an important but under-explored problemmulti-source test-time adaptation from user feedback -where k source models are given, each trained from a distinct source domain, to adapt to a new target domain (figure 1 ). previous work on leveraging multiple knowledge sources is to learn an ensemble @xcite , which means jointly accessing all the models is needed for training. due to its high cost, it is not suitable for real-time updates required by tta. in order to adapt efficiently, we turn this problem into a stochastic decision-making process that trades off model exploration and exploitation. we aim to determine the best adapted model that can perform well in the target domain. we formulate the problem in two frameworks: multi-armed bandit learning and multi-armed dueling bandits @xcite . bandit learning samples one source model each time to receive binary feedback ( or ) ( §4). however, it lacks collaboration among sources and can result in a sub-optimal adapted model. in order not to introduce too much cost, pairwise collaboration between models is explored in dueling bandits @xcite , where two distinct source models are chosen each time for dueling with user preference feedback (e.g., / ) ( §5). a novel method, co-ucb, is proposed to allow collaborative updates. we choose to study the task of extractive question answering (qa), since there are large datasets in different domains that can be used @xcite . more importantly, extractive qa is suitable for eliciting users to leave feedback, since the surrounding context around predicted answer spans can help users to verify the answers. @xcite has simulated user feedback for tta in extractive qa, but not in the multi-source scenario. following previous work @xcite , we simulate user feedback with the annotated answer spans. we conduct our simulation experiments on the mrqa benchmark @xcite , where six domains of extractive qa are studied. we compare the two proposed frameworks to assess their effectiveness and reveal the differences. we also look into the effect of noisy preference feedback. our contributions in this work are as follows: learning from human feedback. human feedback is a useful signal to refine the model outputs and adapt to new domains @xcite , follow human instructions @xcite , etc. human feedback has been explored for different nlp tasks such as machine translation @xcite @xcite , semantic parsing @xcite @xcite , document summarization @xcite , question answering @xcite , and dialogue systems @xcite . in particular, learning from human feedback has gained a lot of interests recently in the context of alignment of large language models (llms) @xcite @xcite . fundamentally, alignment research is necessary and appealing from two aspects: (1) alignment enables a model to go beyond supervised learning @xcite (2) alignment leads to a safer system (ope-nai, 2023). the proposed co-ucb could potentially be used for alignment in future work. who founded the apple computer company? steve wozniak steve wozniak, steve jobs who founded the apple computer company? steve wozniak sample a combination user leaves preference feedback collaborative update sample a model user leaves binary feedback model update co-ucb ucb find bestadapted model figure 2 : the illustration of ucb and co-ucb for multi-armed bandit learning and dueling bandits respectively. for ucb, one model is sampled and receives binary feedback. for co-ucb, two models are chosen each time and preference feedback is made by the user, then the models are updated. ## ucb for bandit learning as illustrated by figure 2 , we apply ucb for multi-armed bandit learning, whose pseudo-code is shown in algorithm 1. action. at each time t, the source model k is selected from k source models which maximizes μk + 2 ln(n ) n k , where μk represents the average reward obtained for the model k by attempting it for n k times, and n is the number of all test data instances received so far. 2 ln(n ) n k represents the confidence interval to the action k, and a larger one means more uncertainty about the action, intending to explore the action more. as training proceeds, the policy becomes more confident about each action. simulated binary feedback ( or ). for each input, the selected model will first predict its answer, then the user leaves the feedback to the prediction. here, we use binary feedback since it is simple enough for the user to provide and has often been used in previous work @xcite . at each time t, a batch input b t ∼ x is obtained for training, which is passed to the model k to obtain the predictions. reward. with a batch of predicted answers, the model k will obtain a vector of simulated reward r @xmath0, 1} |bt| decided by the user. for each data instance in the batch, we follow @xcite to calculate the simulated reward by comparing the predicted answer to the annotated span, where an index-wise exact match is used. if both the predicted start and end positions exactly match the annotated positions, the reward is 1; otherwise, 0. model update. after obtaining the reward, the model k will be updated with a reward-enhanced loss, where the task-specific cross-entropy loss l t (in eq. 1) will be multiplied by the reward r t . inference. after enough iterations, the best adapted model can be found to perform well in the target domain as line 9 of algorithm 1 shows. sampling pairs of models can be formulated as multi-armed dueling bandits (madb) as discussed above. however, previous work on madb only aims to determine the best model @xcite @xcite , so we further propose a novel method which is collaborative ucb (co-ucb) to let a pair of models collaborate, whose pseudo-code is presented in algorithm 2, and illustrated by figure 2 . action. at each time t, with k source models, we construct c k 2 combinations for selection, where each combination is denoted by a pair of model indices ⟨i, j⟩ (@xmath1. the combination ⟨i, j⟩ selected at time t should maximize (μ i + μj )/2 + 2 ln (n )/n i,j , where μi and μj are the average reward obtained up to time t of model i and j respectively, and n i,j is the number of combinations ⟨i, j⟩ explored so far. n is the total number of test data instances received until now. take model i for example. the reward of exploiting model i represents how well model i can beat the other models during dueling. the average reward μi is calculated as follows: obtain the rewards r i and r j for model i and j respectively as in eq. 5; update the models i, j by loss l t as in eq. 7; 11: end for 12: return: k ← arg max j μj + 2 ln(2n )/ k n j,k . in each selection, to calculate the average reward (μ i + μj )/2 for the combination ⟨i, j⟩, we expect ⟨i, j⟩ to be the most worthy action (exploration-andexploitation trade-off), where i and j can mostly beat the rest of the models, which means they are the two strongest models among the k sources so that they can better collaborate to improve them. simulated preference feedback. (e.g., / ) since for each input, the user will receive two answer candidates instead of one, the binary feedback used in bandit learning is not directly applicable. rather than creating new forms of user interaction, we apply preference feedback @xcite @xcite when faced with multiple candidates. since there are only two candidates, leaving preference feedback will be as simple as binary feedback. for the chosen models i and j at time t, the batch of input b t ∼ x will be given to them independently, to obtain the predicted answer spans. then the users need to compare the two predictions to indicate a preference, where the more accurate answer should be picked out. reward. for each data instance in the batch, the reward @xmath2, 1}. @xmath3; @xmath4. this is a strict measurement for preference since the answers without preference are discarded. to simulate the preference, we calculate the quality score of the predicted answers against the annotated spans, where the answer with a higher score would be preferred or no preference is made if the scores are the same. we use the index-wise f1 value as the quality score, which calculates the f1 score over the predicted indices and the annotated indices, so the score is continuous from [0, 1]. for the batch of input b t , the quality score for the model i and j is denoted as a vector s i and s j respectively. the rewards r i and r j for the model i and j respectively are obtained by: collaborative model update. after obtaining the rewards, we perform collaborative model updates. if there is one preferred model, then it will be regarded as the teacher, and its prediction will be used to jointly update the two models. with the predictions from model i and j, we obtain the better one ⟨y * (1) t , y * (2) t ⟩, each as a vector, as: then we jointly update the two models by the loss: models updated in this way can better make use of the benefits of different source models during training, that is, when one model from a selected pair cannot predict a correct answer, the other one may make up for it by sharing its prediction. inference. after adaptation, there is no need to access a pair of models for inference anymore, so we just keep the best performing model by the method in line 12 of algorithm 2. ## conclusion we present the first work on multi-source test-time adaptation from human feedback, where we cast it as an iterative decision-making problem. we first formulate it as multi-armed bandit learning. more importantly, to utilize pairwise collaboration, we further regard it as dueling bandits. co-ucb is a novel method proposed in this work. though we study online adaptation from the online data stream, our work can also be applied in offline model refinement. for the offline setting, we do not update the model online but only update the policy for model selection when receiving user feedback each time. after collecting enough user feedback, we fine-tune the found best model offline with user feedback.