B ayesian Example Selection Improves In-Context Learning for Speech, Text and Visual Modalities (2024)

Abstract: Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of ICL heavily depends on the quality of the in-context examples presented, which makes the in-context example selection approach a critical choice. This paper proposes a novel eBayesian in-Context example Selection method (ByCS) for ICL. Extending the inference probability conditioned on in-context examples based on Bayes’ theorem, ByCS focuses on the inverse inference conditioned on test input. Following the assumption that accurate inverse inference probability (likelihood) will result in accurate inference probability (posterior), in-context examples are selected based on their inverse inference results. Diverse and extensive cross-tasking and cross-modality experiments are performed with speech, text, and image examples. Experimental results show the efficacy and robustness of our ByCS method on various models, tasks and modalities.

## introduction large language models (llms) @xcite have achieved great success on many text-based natural language processing (nlp) tasks. by connecting with extra visual and audio encoders @xcite , the resulting multimodal llms can also achieve remarkable performance on imagetext and audio-text tasks @xcite @xcite . with the ability of incontext learning (icl) @xcite , llms can adapt to new tasks easily and efficiently in a training-free manner, to generate output following the prompting paradigm based on a few input-label pairs pre-pended to the test input. the existence of icl ability has also been verified on image-text and * * corresponding author. this paper proposes bycs (bayesian in-context example selection), a novel in-context example selection approach focusing on mutual information interactions based on the bayesian formula. refer to the inference of test input conditioned on in-context examples as icl inference, and the inference of in-context example's input based on the test input-label pair as the inverse inference. by introducing inverse inference via bayes' theorem, bycs leverages the inverse inference result to evaluate the quality of each in-context example. assuming the contextual information interaction is mutual, an accurate inverse inference is likely to result in an accurate inference. examples with accurate inverse inference results are selected as optimal examples. extensive experiments across audio, image, and text modalities are conducted to verify the effectiveness and robustness of bycs, such as asr, visual question answering (vqa), as well as nlp tasks (including topic classification, sentiment analysis, and text-to-sql etc). our main contributions are summarised as follows: in audio modality, borsos et al. (2023) proposed audiolm, a language model based on quantised speech example inputs speech test input text example labels answer "好睇。" 𝑋 𝐶 !"#@xmath0% text example labels 𝐶 &'()& german. "睇嚟。" image example inputs text example inputs 𝐶 !"#$% text example labels image test input text test input answer 𝐶 &'()& 𝑋 𝑌 does this type of train transport people or cargo? what is this vehicle used for? transporting goods. cargo. (a) text icl (b) asr icl (c) vqa icl figure 2: multimodal icl. although icl on different modalities shares the same formula expression, the actual inputs and inference model architectures differ. for asr icl on whisper, the speech is fed into the encoder while the text example is labelled into the decoder, which is aware of speech input through cross-attention with the encoder. for vqa icl, images are first encoded to the same embedding space of lm's input, then interleaved images and texts are fed into decoder lm. audio tokens for audio generation tasks, which exhibits icl ability for audio continuation. similarly, wang et al. (2023a) proposed vall-e, a controllable text-to-speech synthesis system with icl ability based on audio and text prompts. wang et al. (2023c) presented the first icl work for asr based on paired speech-text examples, which adapted the whisper (radford et al., 2023) model to receive considerable word error rate (wer) reductions on unseen chinese dialects. further explorations enabled the recent speech-language models to perform icl on more speech input tasks through warmup training @xcite or speech instruction-tuning @xcite . in-context example selection methods. @xcite proposed a scoring lm to retrieve incontext examples using contrastive learning, which can also be trained with reinforced learning algorithms, such as q-learning @xcite and policy gradient @xcite . alternatively, examples that are semantically similar to the test input can be selected. @xcite proposed to select the k nearest neighbours (knn) in the embedding space of the examples. when combining with chain-of-thought @xcite , @xcite proposed to select examples in the embedding space of the reasoning path. llm feedback is often used in in-context example selection. figure 3 : the detailed pipeline of our bycs method includes: first, conduct the first-round inference to estimate the label of the test input. then, perform inverse inference on each example in the datastore, where the test input and the estimated label serve as in-context examples. a detailed illustration of inverse inference can be found in figure 5 in the appendix. finally, rank in-context examples by the text similarity between the inverse inference result and the true context label. examples with high similarity scores are selected due to their high mutual information interaction. iter et al. (2023) selected in-context examples with cross-entropy differences of the fine-tuned model based on the assumption that icl may act as implicit gradient descent @xcite . @xcite identified highly impactful examples according to the proposed influence score. although bycs also uses llm feedback when evaluating the quality of in-context examples through inverse inference, it leverages the text-similarity of the inverse inference results and the corresponding ground-truth labels, in no need of complete output probability distributions which are often not available for commercial llms. @xcite selected optimal in-context examples in the bayesian framework by viewing llms as latent variable models and icl as latent concept learning. in comparison, bycs directly extends the icl inference probability using bayes' theorem. @xcite selected examples with high discrepancy between the labels and llm's outputs when performing question answering. bycs also selected examples from candidates in a datastore based on llm's outputs but computes the mutual information interactions between the in-context examples and test input. ## methodology as shown in figure 3 , given a test input x and paired in-context examples (c input , c label ), llms predict the most possible answer ŷ by maximising the inference probability p (y|c input , c label , x): to as text questions and corresponding answers. regarding asr, c input and c label are speech audio and corresponding text transcriptions. regarding vqa, c input are images and text questions based on the images and c label are the text answers. the inference probability can be extended using bayes' theorem: the likelihood p (c label |x, y, c input ) is termed as inverse inference probability, since it can be interpreted as the probability of the context label c label when the test input-label pair (x, y) is inversely treated as the in-context example. bycs is focused on the inverse inference probability and assumes the influence of the prior p (y|x, c input ) is subordinate for simplification. in practice, since the ground-truth label y ref of the test input x is not available, the correct likelihood p (c label |x, y ref , c input ) is approximated by p (c label |x, ŷ, c input ), where ŷ is produced by the first-round inference. specifically, as the text similarity between c label and ĉlabel , and use q as the metric for the evaluation of the quality of inverse inference. since more accurate inverse inference probability often results in higher text similarity, bycs selects the in-context examples with higher q. note that q is adopted since it does not require to assessment of the model's output probability distribution of the llm, which is often unavailable for commercial llms. to reduce the computation cost of inverse inference, two methods are used when the number of examples in the datastore is large: ## conclusion this paper proposes bycs, a novel in-context example selection method based on bayes' theorem, which assumes that contextual information interaction is mutual between the test input and in-context examples and selects high-quality examples based on the inverse inference results. experiments are performed across three modalities: speech, text, and images, using six different tasks and seven datasets. results demonstrated the robustness and effectiveness of bycs. it is also validated that the inverse inference results can be approximated using a smaller model from the same model family, which considerably reduces the computational cost. moreover, relying on text similarity to rank in-context examples, bycs is more suitable for open-ended long-answer datasets which contain sufficient contextual information. future work is to extend the inverse inference to sequences with multiple incontext examples to model the interactions among the in-context examples.