F i D - ICL : A Fusion-in-Decoder Approach for Efficient In-Context Learning (2023)

Abstract: Large pre-trained models are capable of few-shot in-context learning (ICL), i.e., performing a new task by prepending a few demonstrations before the test input. However, the concatenated demonstrations are often excessively long and induce additional computation. Inspired by fusion-in-decoder (FiD) models which efficiently aggregate more passages and thus outperforms concatenation-based models in open-domain QA, we hypothesize that similar techniques can be applied to improve the efficiency and end-task performance of ICL. To verify this, we present a comprehensive study on applying three fusion methods—concatenation-based (early fusion), FiD (intermediate), and ensemble-based (late)—to ICL. We adopt a meta-learning setup where a model is first trained to perform ICL on a mixture of tasks using one selected fusion method, then evaluated on held-out tasks for ICL. Results on 11 held-out tasks show that FiD-ICL matches or outperforms the other two fusion methods. Additionally, we show that FiD-ICL (1) is 10x faster at inference time compared to concat-based and ensemble-based ICL, as we can easily pre-compute the representations of in-context examples and reuse them; (2) enables scaling up to meta-training 3B-sized models, which would fail for concat-based ICL.

## introduction large pre-trained models demonstrated remarkable performance in learning new language tasks via few-shot fine-tuning (ft)-initializing a model with pre-trained weights and optimizing it based on a few examples @xcite . ft-based approaches currently achieve state-of-the-art performance @xcite ), yet they require backpropagating and computing gradients over the full models, which can be prohibitive under memory and resource constraints. concat. raw text ## analysis in this section we evaluate our icl models in various scenarios, in hope to better understand their behavior and limitations. in §6.1 we evaluate the models to perform icl with varying number of shots, when they were originally meta-trained to do 16-shot icl. in §6.2 we study the influence to performance when the in-context examples are perturbed. in §6.3 we try to understand where icl methods lie among other recent advances by comparing the performance of different model families. ## conclusion motivated by the train-test efficiency differences between few-shot in-context learning and few-shot fine-tuning, we aim to find a balance and benefit from the strengths of both approaches. towards this goal, we introduce fid-icl, a fusionin-decoder approach for icl, inspired by fusionin-decoder models for open-domain qa @xcite . with extensive experiments, we show that fusion-in-decoder icl (intermediate fusion) is more favorable compared to concatenation-250m 800m 3b 11b # parameters 40 50 60 70 80 mtest7 avg. accuracy (%) majority/random (a) vs. not metatrained metatrained zeroshot concaticl fidicl ensembleicl tfew ft not metatrained zeroshot concaticl ensembleicl simple ft tfew ft 125m 350m 760m1.3b 2.7b 6.7b 13b 175b # parameters 40 50 60 70 80 mtest7 avg. accuracy (%) majority/random (b) vs. gpt3 models encdec metatrained zeroshot fidicl (#@xmath0, in terms of both performance and computation efficiency. moreover, fusion-in-decoder icl partly closes the gap between gradient-free icl methods and gradient-based fine-tuning methods, highlighting the potential of approximating gradient-based optimization with efficient forwardonly methods @xcite . future work may build upon our insights to further improve the computation efficiency of few-shot learning. however, similar to the findings in @xcite , our analysis on icl models suggest that they barely learn the input-label mapping from the in-context examples. we also have mixed results when more shots become available for the icl model. we hope future work can further improve the performance of icl by enabling it to learn from input-label mapping effectively and faithfully.