How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations (2025)

Abstract: Multimodal foundation models aim to create a unified representation space that abstracts away from surface features like language syntax or modality differences. To investigate this, we study the internal representations of three recent models, analyzing the model activations from semantically equivalent sentences across languages in the text and speech modalities. Our findings reveal that: 1) Cross-modal representations converge over model layers, except in the initial layers specialized at text and speech processing. 2) Length adaptation is crucial for reducing the cross-modal gap between text and speech, although current approaches’ effectiveness is primarily limited to high-resource languages. 3) Speech exhibits larger cross-lingual differences than text. 4) For models not explicitly trained for modality-agnostic representations, the modality gap is more prominent than the language gap.

## introduction recent progress in foundation models has sparked growing interest in expanding their text processing capabilities @xcite @xcite to speech @xcite @xcite . despite the empirical successes, understandings of these models' internal representations remain limited, particularly on language differences, modality gaps, and the impact of model architectures. this work aims to fill this gap by studying how text and speech are represented in recent multimodal foundation models. while the internal representations of multilingual models have been extensively studied, most prior works focus on single-modality analyses of text @xcite or speech @xcite @xcite @xcite . moreover, as many multimodal foundation models have dedicated subparts for languages or modalities, not all analysis techniques are directly applicable. for instance, similarity retrieval tasks @xcite @xcite often require identical input feature dimensions, which is not always guaranteed for speech and text. probing @xcite @xcite features with different dimensions leads to auxiliary classifiers of varying sizes and may skew the results. in this work, we use singular vector canonical correlation analysis (svcca; @xcite due to its invariance to affine transformations, which is suitable for comparing features from different architectures and dimensions that occurs frequently in speech-text representations. previous studies comparing speech-text representations do not involve the cross-lingual aspect, and use either task-specific @xcite or proprietary @xcite models. recent studies on multilingual representations in large language models reveal different levels of language invariance depending on training data @xcite and model scales @xcite . the study most related to ours is probably the concurrent work from @xcite , who show that representations for semantically equivalent multilingual/modal inputs are similar in model intermediate layers. our study differs in its focus on languages and modality gaps as well as its broad coverage of 30 languages at differ-ent resource levels. to the best of our knowledge, we present the first cross-modal and cross-lingual analysis of representations over a wide variety of language in multimodal foundation models. ## methodology an assumption in unified multimodal and multilingual models is that inputs are transformed into a semantic space independent of input forms. this abstraction from surface level motivates our method. measuring similarity between semantically equivalent sentences: as shown in figure 1 , we begin with semantically equivalent sentences in different languages and modalities. to compare their model activations at different layers, we extract these activations and employ svcca. its invariance to affine transformations @xcite ensures comparabilitiy of activations across different modalities and languages, even when they originate from different model subparts. given the extracted activations, we calculate the svcca scores between speech and text versions of the same sentence (intra-lingual cross-modal) and between translations (cross-lingual text/speech). higher svcca scores indicate higher similarity. more explanations of svcca scores are in appendix a. ## results we analyze cross-modal ( §3.1) and cross-lingual ( §3.2) representations, and compare the impact of modality and language differences ( §3.3). ## conclusion to study how multimodal foundation models process text and speech across diverse languages, we analyzed their internal representations based on the similarity of semantically equivalent sentences. our findings highlight that while these models present a unified architecture for handling various modalities and languages, they do not inherently create fully unified representations by semantic meaning. representational gaps, some of which already observed in task-specific models, including speech-text length mismatches (e.g., @xcite , weak representation for low-resource languages, and tokenizer bottleneck (e.g., @xcite , still persist in current multimodal foundation models. besides the findings presented earlier, our study offers several practical recommendations. the first is to incorporate representation analyses into the development cycle of models, especially on models designed to reduce modality gaps. another recommendation is model choices for speech-text downstream tasks. practitioners working with lowresource or zero-shot use cases may consider initializing their models with foundation models explicitly trained for closing modality and language gaps.