Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison (2025)

Abstract: Following the remarkable success of Large Language Models (LLMs) in NLP tasks, there is increasing interest in extending their capabilities to speech—the most common form of communication. The most widespread approach to integrating speech into LLMs is dense feature prepending (DFP), which prepends the projected speech representations to the textual representations, allowing end-to-end training with a speech encoder. This raises questions about the need for a sophisticated speech encoder for DFP and how its performance compares with a standard encoder-decoder (i.e., cross-attention) architecture. We compare DFP and cross-attention under a variety of configurations, such as CTC compression, sequence-level knowledge distillation, on monolingual, bilingual, and multilingual models. To perform a controlled architectural comparison, we train all models from scratch rather than using large pretrained models and use comparable data and parameter settings, testing speech-to-text recognition (ASR) and translation (ST) on MuST-C v1.0 and CoVoST2 datasets. Despite the wide adoption of DFP, our results do not indicate a clear advantage of DFP over cross-attention.

## introduction as the nlp community has witnessed the emergence of large language models (llms) and their remarkable performance in tackling nlp tasks @xcite @xcite , there is increasing interest in extending their capabilities to other modalities, such as audio @xcite @xcite and images @xcite , to broaden their applicability. one of the most natural extensions of llms is to inject them with speech -the most common form through which humans express their language @xcite -to exploit the † these authors contributed equally to this work llm's linguistic fluency and skills to tackle speechto-text (s2t) tasks, such as automatic speech recognition (asr) and s2t translation (st). this goal has been predominantly pursued by dense feature prepending (dfp) which adapts the embedded speech representations -obtained using the encoder of a speech foundation model (sfm) or a speech encoder trained from scratch -to the input feature space of an llm via a modality adapter and, optionally, a length adapter @xcite @xcite and prepends them to a textual prompt describing the tasks to be performed @xcite . existing works on dfp mainly train the speech encoder, the modality adapter, and low-rank adapters in the llm in an end-to-end fashion @xcite . throughout the paper, we refer to this dfp solution leveraging a speech encoder or sfm as decoder-prepend. the effectiveness of decoder-prepend has recently been questioned on the basis of the analogy with classic s2t encoder-decoder models @xcite , where the integration of the encoder output into the decoder is performed through cross-attention modules @xcite @xcite . in addition, the outstanding performance of decoder-only llms on nlp tasks traditionally handled by encoder-decoder models has motivated the exploration of decoder-only s2t models @xcite , which can be regarded as dfp solutions that question the need for a speech encoder and directly prepend speech features to the text embeddings. in this context, @xcite ; @xcite highlighted the crucial role of relaxing the causal 1 masking in the self-attention modules typical of autoregressive models for the speech features, allowing them to look at each other freely, rather than being restricted to only previous elements. notably, @xcite claimed that this approach enables decoder-only models to even surpass encoder-decoder ones on the asr task. on the contrary, to the best of our knowledge, no investigation on the effect of the causality property has been carried out for decoder-prepend models. with the goal of shedding light on the strengths and weaknesses of dfp solutions, we i) systematically compare the two dfp-based architectures (decoder-prepend and decoder-only) with the standard encoder-decoder architecture using crossattention, not only in terms of performance but also computational demands; ii) perform this comparison under a variety of relevant configurations, covering monolingual, bilingual and multilingual settings and including widely adopted techniques such as speech sequence length reduction (using the ctc compression mechanism - @xcite ; @xcite ) and sequence-level knowledge distillation or seqkd @xcite ; and iii) conduct an in-depth study of the causality properties of dfp architectures. to ensure a sound and fair comparison across architectures, we train them from scratch on the same two datasets: must-c v1.0 @xcite and covost2 @xcite . this choice of not using large-scale pretrained models offers a number of advantages. first, it prevents our results from being influenced by i) the specific features and capabilities of pretrained models @xcite , and ii) the lack of a well-established method for integrating speech encoder with llms via cross-attention, an area still in its early research stages @xcite . furthermore, using small models instead of largescale ones allows us to investigate the impact of all the aforementioned configurations within a reasonable computational budget. our experiments, carried out on two s2t tasks -asr and st -and covering a total of 12 language directions, demonstrate that: while the scalability of the findings to largescale models has to be confirmed (see limitations), we believe that these findings can inform future research on integrating dense speech features into llm. we release the code used in our experiments under the apache 2.0 license at: https: //github.com/hlt-mt/fbk-fairseq/ . ## results in this section, we conduct a series of experiments to examine cross-attention and dfp from a variety of angles under comparable data and model size conditions. to begin with, we first discuss the asr and st results between cross-attention, decoderonly, and decoder-prepend using transformer and conformer architectures. then, we present their results under the effect of speech sequence compression (via ctc) and seqkd. in addition, we analyze their difference in terms of generation speed and gpu memory footprint. finally, we present an ablation study about the causality masking of decoder-only and decoder-prepend. ## conclusion in this paper, we aim to validate the modeling choice of using dfp (decoder-only and decoderprepend) over cross-attention to integrate speech into decoder-only llms for s2t tasks. in order to perform a controlled comparison under limited computational budget, we train all models from scratch without using large pretrained models. our series of comparisons, including mono/bi/multilingual settings, indicate that dfp does not consistently outperform cross-attention in asr and st quality, and that cross-attention is more efficient in terms of generation speed and gpu memory footprint. our studies further suggest that: (1) decoder-prepend with a strong speech encoder is more efficient than decoder-only of similar size, and (2) a variety of test sets, language pairs (and directions) as well as tasks, e.g., bi/multi-lingual st models, are needed to validate the effective scope of a s2t technique, such as causal masking. future work. in addition to scaling up the data and model sizes, we leave the comparisons of several interesting aspects of s2t models in future works. these include 1) zero-shot transfer (tsiamas et al., 2024), 2) performance under segmental inputs and augmentations @xcite @xcite , 3) simultaneous setting @xcite and 4) additional tasks such as spoken language understanding @xcite and spoken question answering @xcite .