BTW : A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration (2025)

Abstract: Mixture-of-Experts (MoE) models have become increasingly powerful in multimodal learning by enabling modular specialization across modalities. However, their effectiveness remains unclear when additional modalities introduce more noise than complementary information. Existing approaches, such as the Partial Information Decomposition, struggle to scale beyond two modalities and lack the resolution needed for instance-level control. We propose **B**eyond **T**wo-modality **W**eighting (**BTW**), a bi-level, non-parametric weighting framework that combines instance-level Kullback-Leibler (KL) divergence and modality-level mutual information (MI) to dynamically adjust modality importance during training. Our method does not require additional parameters and can be applied to an arbitrary number of modalities. Specifically, BTW computes per-example KL weights by measuring the divergence between each unimodal and the current multimodal prediction, and modality-wide MI weights by estimating global alignment between unimodal and multimodal outputs. Extensive experiments on sentiment regression and clinical classification demonstrate that our method significantly improves regression performance and multiclass classification accuracy.

## introduction multimodal learning has advanced rapidly in vision-language reasoning @xcite , emotion recognition @xcite , and clinical decision support @xcite . sparsely-gated mixture-of-experts (moe) models have emerged as powerful and efficient solutions for scaling multimodal architectures through modular specialization across expert subnetworks @xcite . designed for modality-specific and cross-modal patterns, moe models have achieved success in multimodal fusion, including visionlanguage grounding, representation learning, and alignment @xcite . recent moe frameworks flexibly integrate three or more modalities, achieving strong results across domains @xcite @xcite . despite these advances, it is unclear if integrating multiple modalities adds more noise than useful information, or if current moe designs capture cross-modal interactions effectively. as shown in figure 1 , the text-only model outperforms the full multimodal system in 5-class and binary sentiment classification (acc-5, acc-2), as well as weighted-f1, on the cmu-mosi dataset @xcite . this result suggests that merely aggregating modalities can hurt performance when one view is substantially more informative, as also seen in healthcare @xcite , highlighting the need for a mechanism that both stabilizes variance and selectively amplifies complementary information to achieve a more effective multimodal integration. although multiple approaches for modeling modality interactions exist ( to address these limitations, we propose btw, a novel bi-level weighting mechanism that stabilizes variance across modalities and manages noise from added modalities. at the instance-level, btw employs kullback-leibler (kl) divergence @xcite to measure how much multimodal predictions capture the distributional information from each individual modality. at the modality-level, it leverages mi to quantify global reliability and contribution of each modality's prediction across the dataset. by combining kl and mi, btw dynamically balances instance-level contributions with global modality reliability, reducing variance while preserving complementary information. based on experimental results, our framework improves model stability and performance across both continuous (regression) and categorical (classification) tasks, as demonstrated by performance gains in emotion recognition tasks @xcite , @xcite ) and clinical length-ofstay prediction tasks @xcite . ## method our proposed btw framework, as shown in figure 2 , specifically addresses the need for a generalizable solution capable of scaling to arbitrary modalities by systematically analyzing modality interactions while simultaneously stabilizing variance. the framework is built on top of existing moe models @xcite and involves three steps: (1) obtaining unimodal predictions, (2) computing bi-level weights, and (3) dynamically applying these weights to modality embeddings. ## experiment we conduct experiments to validate btw in stabilizing variance and evaluating robustness across modalities, tasks, and diverse domains. details on computation, time cost, and hyper-parameters are in appendix f. ## ablation study 6.1 weighting mechanisms to better understand the individual contributions of each weighting component in our btw framework, we conducted ablation experiments on the sentiment analysis task. specifically, for btwglobal (kl), we modify eq. 8 by replacing the instance-level weight for each modality with its average across the dataset, thus using a constant modality-specific weight for all samples. for btwglobal (mi), we make an analogous change to eq. 9, using only the global mi value for each modality and omitting instance-level weights. btw-global (mi) consistently outperforms btw-global (kl), highlighting the effectiveness of mi in capturing global modality alignment. compared with btw-global (kl), btw-local (kl) demonstrates a clear advantage of instance-level weighting across nearly all metrics on both datasets, underscoring the benefits of fine-grained variance stabilization at the instance-level. lastly, btw reveals its complementary strengths by consistently outperforming the btw-global (mi) in multiclassclassification accuracy (e.g., cmu-mosei acc-7: 52.62% vs. 52.34%), indicating the synergy between local variance management and global modality informativeness. in summary, kl provides effective instancelevel variance control beneficial to regression, while mi contributes significantly to global alignment and multi-class accuracy. the bi-level weights yields balanced, robust multimodal fusion. ## conclusion this paper introduces a bi-level, non-parametric weighting framework that advances multimodal learning beyond two modalities by addressing prediction variance and modality interaction explainability. by integrating instance-level kl divergence with modality-level mutual information, the method adaptively calibrates modality contributions without introducing additional trainable parameters. the framework's value lies not only in quantitative improvements but also in its efficiency as a non-parametric, plug-and-play module that enhances existing architectures. extensive experiments on diverse benchmarks and both regression and classification tasks demonstrate how the framework stabilizes variance in high-density regions and resolves ambiguity by leveraging globally informative modalities, ultimately facilitating more robust and transparent multimodal models.