Unleashing the Potentials of Likelihood Composition for Multi-modal Language Models (2024)

Abstract: Model fusing has always been an important topic, especially in an era where large language models (LLM) and multi-modal language models (MLM) with different architectures, parameter sizes and training pipelines, are being created all the time. In this work, we propose a post-hoc framework, aiming at fusing heterogeneous models off-the-shell, which we call likelihood composition , and the basic idea is to compose multiple models’ likelihood distribution when doing a multi-choice visual-question-answering task. Here the core concept, likelihood , is actually the log-probability of the candidate answer. In likelihood composition , we introduce some basic operations: debias , highlight , majority-vote and ensemble . By combining (composing) these basic elements, we get the mixed composition methods: mix-composition . Through conducting comprehensive experiments on 9 VQA datasets and 10 MLMs, we prove the effectiveness of mix-composition compared with simple ensemble or majority-vote methods. In this framework, people can propose new basic composition methods and combine them to get the new mixed composition methods. We hope our proposed likelihood composition can provide a new perspective of fusing heterogeneous models and inspire the exploration under this framework.

## introduction recently numerous multi-modal language models are emerging, e.g., llava @xcite , minigpt4 @xcite , blip-2 @xcite , qwen-vl @xcite , internvl @xcite , and sphinx @xcite , each characterized by different architectures, parameter sizes, training datasets, and pipelines. consequently, these models exhibit varying strengths across different tasks 1 code is released zhaoshitian/likelihood-composition-toolkit and domains. some works @xcite @xcite have demonstrated that fusing multiple models can enhance performance and generalizability across diverse domains. thus, several model fusion techniques have been devised to leverage the complementary capabilities of these models. many works focus on getting a new model by inheriting the knowledge from multiple parent models. some of them interpolate several models' weights to get the new model's weight, e.g., wise-ft @xcite and model soup @xcite . however, in this process, all the parent models and the new derived model need to have the same architecture and parameter sizes, i.e., in most cases, the parent models are the finetuning versions of one pretrained model, leading to lack of diversity of these models. there are also some works focusing on distilling the knowledge from several different parent models @xcite . however, the training computation cost makes it hard for researchers to combine parent models freely, i.e., the computation cost of the distillation training process limits researchers to do lots of trial-and-error experiments to get a good parent models recipe. considering these issues, a promising line of work @xcite @xcite fuse different models via manipulating or composing their likelihood distributions foot_0 , with the advantage of being fully post-hoc, training-free and the same architecture of parent models is not necessary. the basic operation is to average all models' likelihood distribution of the candidate answers, called "ensemble". @xcite recently some works also tried to combine the likelihood distributions from one model with different prompts as input @xcite @xcite . behind these works, the core concept is to fuse different models via combining their likelihood distribution of the candidate answers, to boost the performance of downstream tasks, specifically multichoice vqa tasks. thus, we propose a framework named likelihood composition. under this framework, we introduce some basic operations: debias, highlight, ensemble and majority-vote, some of which have been proposed in the previous research @xcite @xcite . and these basic operations are classified into two classes: "self-composition" and "mutual-composition". by composing these basic operations, we derive some new likelihood composition methods, e.g., ensemble-debias, ensemblehighlight, majority-debias and majority-highlight, which we call "mix-composition". to explore the likelihood composition's effectiveness on fusing models and boosting the performance on downstream tasks, we conduct the experiments on llava series and other 4 advanced multi-modal language models and 9 vqa datasets. our experiment results reveal some interesting findings: model ensemble to boost the performance on downstream tasks, a usual method is to ensemble multiple models. in this literature, many lines of research are developed: weight interpolation @xcite , model collaboration @xcite @xcite and distillation-based methods @xcite . decoding methods for language models given a pretrained large language model, the decoding method also matters a lot. some works focus on the decoding techniques of llm, e.g., contrastive decoding(o'brien and lewis, 2023), contrasting a pair of weak and strong llm's logits to improve the strong model's generation quality; proxy tuning @xcite , doing arithmetic among three pretrained llms to boost the generation ability. ## preliminary before introducing the detailed methodology of likelihood composition officially, it is necessary to make clear some core concepts that may be mentioned in the methodology part. so in this section, we give a clear description of the task formulation, which is the basic setting of our study, and an accurate definition of "likelihood", the core concept used in our framework. task formulation considering most multi-modal tasks, e.g., visual grounding and image retrieval can be formulated as a visual-question-answering task @xcite . in this paper, we exclusively investigate multi-choice visual question answering (vqa) tasks. the task formulation is as follows: given a dataset @xmath0, s @xmath1, q i , c i ) consists of an image i i and a question q i . the candidate answers for this image-question pair are represented as c i , a list containing n candidate answers: c @xmath2, ..., c i n ]. and we need to input (i i , q i , c i ) into mlm to predict the right answer's option letter. likelihood calculation for a vqa sample (i, q, c), during the normal forward process, both i and q are input into the model. c is a list of choices, denoted as [c 0 , c 1 , ..., c n ], where n is the number of choices. the likelihood of candidate c i is calculated as follows: once the likelihood of each choice is calculated, we denote the list containing all the choices' likelihood values as y, upon which composition is performed. 34 4 likelihood composition first, we organize the basic operations in the likelihood composition framework into two categories: self-composition and mutual-composition, as illustrated in fig. 1 . then we mix these basic elements, resulting in the mix-composition. ## experiments to evaluate the efficiency of likelihood composition methods on improving mlm's performance on vqa tasks, we conduct experiments on 10 advanced multi-modal language models and 9 vqa benchmarks. ## conclusion in this work, we propose "likelihood composition", a framework unifying some operation in the model fusing field. based on this framework, we further propose "mix-composition", mixing the "selfcomposition" and "mutual-composition". in our experiments, we find "self-composition" can boost the mlm significantly on vqa tasks and "mixcomposition" also bring significant improvement compared with "mutual-composition".