Imagine to Hear: Auditory Knowledge Generation can be an Effective Assistant for Language Models (2025)

Abstract: Language models pretrained on text-only corpora often struggle with tasks that require auditory commonsense knowledge.Previous work addresses this problem by augmenting the language model to retrieve knowledge from external audio databases.This approach has several limitations, such as the potential lack of relevant audio in databases and the high costs associated with constructing the databases. To address these issues, we propose Imagine to Hear, a novel approach that dynamically generates auditory knowledge using generative models. Our framework detects multiple audio-related textual spans from the given prompt and generates corresponding auditory knowledge. We develop several mechanisms to efficiently process multiple auditory knowledge, including a CLAP-based rejection sampler and a language-audio fusion module. Our experiments show that our method achieves state-of-the-art performance on AuditoryBench without relying on external databases, highlighting the effectiveness of our generation-based approach.

## introduction can language models (lms) understand auditory signals like humans? recent studies suggest that the answer is negative. although lms do exhibit some understanding of audio signals @xcite , they perform poorly in answering questions that require auditory commonsense knowledge @xcite . for example, lms pretrained on textonly datasets do not know which object is more likely to make a higher-pitched sound or which animal is most relevant to the given onomatopoeia. while such limitations can be partially addressed by augmenting language models with auditory representations retrieved from external audio database @xcite , this approach suffers from two * equal contribution † corresponding author key limitations: (1) the database may not contain any audio that is directly relevant to the given query. to address these challenges, we present a novel approach, coined imagine to hear (ith), that leverages the imaginative capabilities of audio generative models @xcite @xcite . that is, ith acquires auditory knowledge directly related to the given task by generating it instead of retrieving it from a database. ith works in three steps. first, we extract textual spans from the given input prompt that may contain auditory knowledge. unlike visual imagination literature @xcite , we extract multiple short spans instead of a single long span so that we can generate cleaner audio without auditory interference. next, we generate audio knowledge for each span, ensuring their relevance via clapbased rejection sampling @xcite . finally, we inject generated audio into the lm with a newly developed fusion module that can efficiently handle an arbitrary number of auditory features. the proposed ith achieves state-of-the-art results on auditorybench @xcite , an auditory commonsense benchmark. the primary contributions of our work can be summarized as follows: the sound of cello is [mask] than the sound of violin classifier self-attention layer feed-forward network x n language encoder mlp diffusion ast encoder clap span detector x n imagination module σ w t w a e at e t + × × + ~fusion layer feed-forward network fusion gate unit fusion module ... t 2d t nd t 1d ... t 22 t n2 t 12 ... t 21 t n1 t 11 t (n-1)d t (n-2)d t nd ... t (n-2)2 t n2 t (n-1)1 t (n-2)1 t n1 ... sound of violin ... a 2 a n ... a (d-1) a (d-2) a d ... ... a 2 a 1 ... t 2d t nd t 1d ... t 22 t n2 t 12 ... t 21 t n1 t 11 .... key query attention mask a 3 a 2 a 1 t 3d t 2d t 4d t 1d t 32 t 22 t 42 t 12 t 31 t 21 t 41 t 11 .... the sound of cello ... a 2 a n ... a (d-1) a (d-2) a d ... ... a 2 a 1 value a 3 a 2 a 1 t (n-1)2 ... ... ## method: imagine to hear the proposed ith consists of three components (figure 1 ): (1) imagination module for audio feature generation (section 2.1); (2) fusion module for multimodal integration (section 2.2); (3) language encoder for text feature generation (section 2.3). ## experiments we conducted our experiments on the auditory-bench with further experiments. more details are provided in appendix d. ## conclusion in this work, we proposed imagine to hear, a novel generation-based approach for injecting auditory knowledge into language models. by dynamically generating audio, our method overcomes the limitations of retrieval-based systems. with a hierarchical architecture and a rejection system, we ensure high-quality, context-relevant audio knowledge integration. experiments demonstrate significant improvements on auditorybench with audio imagination. this work underscores the value of imagination in enhancing multimodal capabilities in language models. ith extends beyond its established advantages over retrieval methods in the multimodal era. its utility is notable in resource-constrained settings reliant on unimodal models where multimodal inputs are infeasible. furthermore, even for advanced multimodal systems, ith can provide a practical pathway when audio is absent, ambiguous, or subtle-parallel to visual imagination modules that enhance reasoning without direct visual input @xcite @xcite .we expect that our approach will remain effective and complementary in the era of multimodal language models.