Let’s Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLM s for Robust and Instruction-Aware ASR and OCR (2025)

Abstract: We introduce “Generative Fusion Decoding” (GFD), a novel shallow fusion framework, utilized to integrate large language models(LLMs) into cross-modal text recognition systems inlculding automatic speech recognition (ASR) and optical character recognition (OCR). We derive the formulas necessary to enable GFD to operate across mismatched token spaces of different models by calculating likelihood at the byte level, thereby enabling seamless fusion and synchronous progression during the decoding process. GFD is plug-and-play bydesign, making it readily compatible with various auto-regressive models without the need for any re-training. GFD proves effective for general ASR and OCR tasks through intermediate and frequent interactions with LLMs, surpassing cascaded methods in English and Mandarin benchmarks. In addition, GFD transfers in-context learning abilities of LLMs and allows for adaptive ASR in instruction-aware andlong-context settings, yielding significant WER reductions of up to 17.7%.

## introduction integrating large language models (llms) into multi-modal systems has recently emerged as a frontier, significantly advancing applications such as automatic speech recognition (asr) @xcite , visual question answering (vqa) @xcite , and reinforcement learning @xcite . despite their robust capabilities, integrating llms with text recognition systems like asr and ocr poses challenges due to the need for high-quality paired data and extensive training resources. modern llms are trained on trillions of text tokens @xcite , far exceeding the data used for end-to-end asr or ocr models @xcite . various fusion strategies have been explored in asr literature, including shallow fusion @xcite @xcite , late fusion @xcite , mid fusion @xcite , and early fusion @xcite . however, these methods face challenges such as discarding the asr decoder's denoising abilities @xcite and requiring aligned token spaces. volatility of model from further training is also a concern when dealing with extensively trained models. to address these challenges, we introduce a novel shallow fusion framework called "generative fusion decoding" (gfd). gfd operates across mismatched token spaces by calculating likelihood at the byte level, enabling seamless integration of llms with text recognition models during the synchronous decoding process (section 3.1). this plug-and-play framework allows llms to correct text recognition errors in real-time (section 3.2), broadening the exploration space and improving recognition accuracy. empirically, gfd is effective on general asr, especially in challenging scenarios like homophones in mandarin and code-switching @xcite (section 4.2). in addition, gfd transfers long-context awareness and in-context learning @xcite of llms and allows for adaptive asr. gfd maintains semantic consistency in long-form audio by leveraging transcription history for contextual biasing (section 4.3). by using controlling prompts such as domain tags, rare words, and explicit instructions, domain sensitivity and instruction awareness is exhibited across various benchmarks (section 4.4). to the best of our knowledge, this unique aspect of llm integration has not been reported in prior work @xcite @xcite . further comparisons with existing methods are discussed in section 5. a key challenge in this integration lies in the mismatch between the token spaces of the two model types, which prevents direct fusion. to address this, we derive a formulation (section 3.1, equation 7, equation 8 ) that enables gfd to compute likelihoods at the byte level, allowing effective fusion during the decoding stage. here, z tr and z lm denote the contextual information from the text-recognition model and the llm, respectively. ## conclusion in summary, we propose generative fusion decoding (gfd), a simple yet effective framework for integrating large language models into asr and ocr systems through byte-level shallow fusion. our theoretical derivations provide a foundation for this integration, while empirical results demonstrate consistent gains across diverse conditions-including noisy audio, long-form inputs, and instruction-following tasks. gfd also compares favorably with prior fusion methods, highlighting its potential as a general-purpose solution. these results establish a foundation for future exploration of fusion methods that further exploit the strengths of pre-trained language models.